University of Salford
Information Systems Institute
BSc Business Information Systems
Spreadsheet Auditing
Final Year Project
Final Report
David Nixon
Supervisor: Mike O’Hara
Submission: 3rd May 2001
Contents
Contents............................................................................................. 2
Abstract.............................................................................................. 4
Acknowledgements ............................................................................ 5
Introduction ....................................................................................... 6
Spreadsheet History ........................................................................... 8
Academic Research.......................................................................... 10
Why Worry About Spreadsheet Errors? ..............................................................10
Spreadsheets can be hazardous to your health [Ditlea 1987]...........................10
Re: ’Computer Error’ in Durham N.C. Election Results [Woodburg 1989] ...10
Computer error costs graduate job [The Daily Telegraph 23/7/98] .................10
How Personal Computers Can Trip Up Executives [Business Week 24/9/84] 10
Netfuture on-line newsletter (1997) .................................................................11
How Frequently Do Spreadsheet Errors Occur?..................................................11
Hitting the Wall: Errors in Developing and De-bugging a ‘Simple’
Spreadsheet Model [Panko 1996] ....................................................................11
The Subversive Spreadsheet [Butler 1997] ......................................................12
What We Know About Spreadsheet Errors [Panko 1998]...............................12
A Taxonomy of Error Types [Panko & Halverson 1996] ................................14
Classification of Spreadsheet Errors [Rajalingham et al 2000] .......................16
A Survey of Research on Spreadsheet Risks [Panko & Halverson 1996].......17
Two Corpuses of Spreadsheet Errors [Panko 2000] ........................................17
What Can Be Done To Avoid Spreadsheet Errors? .............................................18
An Approach to the Teaching of Spreadsheets using Software Engineering
Concepts [Chadwick et al 1999] ......................................................................18
What Constitutes Good Spreadsheet Design?......................................................19
The Subversive Spreadsheet [Butler 1997] ......................................................19
Spreadsheets [Binning 1999] ...........................................................................20
Modelling Optimisation Problems In The Unstructured World Of Spreadsheets
[Ragsdale & Conway 1997] .............................................................................20
Quality Control in Spreadsheets: A Software Engineering-Based Approach to
Spreadsheet Development [Rajalingham et al 2000] .......................................20
A Structured Methodology for Spreadsheet Modelling [Knight et al 2000] ...21
Efficient Methods for Checking Integrity: An Integrated Spreadsheet
Engineering Methodology (ISEM) [Rajalingham et al 1999) .........................21
Spreadshe et Modelling Best Practice [Reed & Batson 1999] .........................22
How Can We Audit Spreadsheets? ......................................................................23
Risk Assessment for Spreadsheet Developments: Choosing which Models to
Audit [Butler 2000] ..........................................................................................23
Integrity Control of Spreadsheets: Organisation and Tools [Rajalingham &
Chadwick 1998] ...............................................................................................24
Visual Checking of Spreadsheets [Chen & Chan 2000a] ................................24
Quality Control in Spreadsheets: A Visual Approach using Colour Coding To
Reduce Errors in Formulae [Chadwick et al 2000] .........................................25
Applying Code Inspection to Spreadsheet Testing [Panko 1999] ...................25
Are Two Heads Better Than One? (At Reducing Errors in Spreadsheet
Modelling) [Panko & Halverson 1996c] ..........................................................26
2
Spreadsheet Errors: What We Know. What We Think We Can Do [Panko
2000] ................................................................................................................26
Trends in Academic Research into Spreadsheets.............................. 27
Academic Research Conclusions ..................................................... 28
Software Spreadsheet Auditing Tools.............................................. 29
The Sample Errors Spreadsheet....................................................... 30
Method Used to Test Software Tools ............................................... 33
Excel’s Built In Auditing functions ................................................. 35
The Spreadsheet Detective .............................................................. 37
The Excel Auditor .......................................................................... 40
The Operis Analysis Kit .................................................................. 43
Spreadsheet Audit for Customs and Excise (SpACE)....................... 45
Results ........................................................................................... 47
Results by Software Tools ...................................................................................47
Results by Type of Error ......................................................................................49
Results by Individual Errors ................................................................................50
Test Conclusions ............................................................................ 53
Overall Conclusions ......................................................................... 54
Conclusions From Project....................................................................................54
Personal reflections..............................................................................................54
Areas For Further Study ................................................................. 58
Appendix A – References................................................................. 59
World Wide Web Pages.................................................................. 59
Published/Academic Documents ..................................................... 59
Appendix B – The Sample Errors Spreadsheet................................ 63
Appendix C – The Spreadsheet Auditing Software Test Form........ 70
Appendix D – Test Results and Screenshots .................................... 73
Appendix E – Project Proposal.......................................................112
Appendix G – Call For Papers for EuSpRIG Conference ..............126
Appendix H – Plagiarism Statement...............................................127
3
Abstract
It is now widely accepted that errors in spreadsheets are both common and potentially
dangerous. Further research has taken place to investigate how frequently these errors
occur, what impact they have, how the risk of spreadsheet errors can be reduced by
following spreadsheet design guidelines and methodologies, and how effective
auditing of a spreadsheet is in the detection of these errors.
This report is divided into two sections. The first summarises the research to date into
spreadsheet errors, the second documents a practical test of five software tools
designed to assist in the audit of spreadsheets. The sec ond section includes the
experiments performed on the five software tools investigated and an investigation
into how these tools work, and concludes that whilst software tools are undoubtedly
useful in the detection of spreadsheet errors, they have much more success on some
errors than they do on others.
4
Acknowledgements
The author would like to thank the following for their support and assistanace:
John Dooley, Colin Ferguson and Andrew Spendlove at CWS for their continued
support throughout the time spent studying the degree
Mike O’Hara, for his support and advice throughout this project.
Ray Butler for his general advice and for his assistance in the SpACE test.
Andy Wiggins at BYG Software for the copy of The Excel Auditor.
Southern Cross Software for the evaluation copy of The Spreadsheet Detective.
David Colver at Operis for the evaluation copy of Operis Analysis Kit.
Emma Clegg and Lianne Nixon, for proof-reading this report.
5
Introduction
Spreadsheets are widely accepted as the second most commonly used PC applications.
Modern spreadsheets are powerful enough to enable complex calculations to be
designed and processed by non-technical users. Unfortunately, the apparent simplicity
of spreadsheets has contributed to the emergence of spreadsheet errors as a serious
problem.
This report documents an investigation into the research that has taken place into
spreadsheet errors and a test of various software tools designed to assist in the
detection of these errors. This report is the conclusio n of a project initiated with a
Project Proposal and revised in an Interim Report (both included in the Appendices).
The primary objectives of this project, as documented in the Project proposal and
Interim Report, are as follows:
•
•
•
•
•
Investigate, summarise and analyse the research to date on spreadsheet errors.
Investigate, summarise and analyse research into how developers try to avoid
software errors from occurring.
Investigate, summarise and analyse how spreadsheets are audited.
Identify and review the sof tware available to assist in spreadsheet auditing.
Apply the spreadsheet auditing methods and software tools identified to either a
standard custom built spreadsheet, or a series of spreadsheet models to investigate
their impact on error detection/correction.
The software tools testing section was designed to answer the following questions:
•
•
•
•
Are software-auditing tools for spreadsheets useful?
How do software-auditing tools assist the user in auditing spreadsheets?
What errors are software-auditing tools good at identifying?
What errors are software-auditing tools poor at identifying?
In order to answer these questions, software tools were tested against a sample errors
spreadsheet, which contained 17 seeded errors. In the main, these tools followed a 3stage format, incorporating familiarisation with the software tools, the test against the
Sample Errors Spreadsheet itself, and the analysis of the first two practical steps.
The body of this report is divided into two sections. The first section involves
research that has taken place into spreadsheet errors, particularly the incidence and
impact of errors, the classification of errors, avoiding errors by following good
spreadsheet design, and the similarities between spreadsheet development and
software engineering. This section also includes an investigation into the trends
followed by spreadsheet researchers since the early 1980s. The second section deals
with the test itself, beginning with an introduction to the Sample Errors Spreadsheet
and the software tools used in the test, and going on to document how each software
tool performed in the tests. This is followed by an analysis of the test results by
software tool, by type of error and by individual error.
6
In the early stages of this project, the author was invited to submit a paper for
consideration for inclusion in the EuSpRIG conference in Amsterdam in July 2001.
The author has accepted this invitation and it is anticipated that the paper submitted
will be primarily based on the second part of this report.
7
Spreadsheet History
The spreadsheet is now the second most popular application in use on PCs. Only the
word processor can claim more usage. It is undeniable that spreadsheets have had a
major impact on the popularity of Personal Computers over the last 20 years.
Perhaps the main reasons that spreadsheets have become so popular in business are
the apparent simplicity and flexibility available in spreadsheets. Spreadsheets are
often considered simple; as the training needed to begin using them can be minimal,
allowing even non-technical users to achieve results. The production of spreadsheet
models allow even these non-technical users to use pre-defined formulae and
procedures set up within a model. Spreadsheets are also flexible enough to perf orm
tasks, albeit in a rather less efficient manner, that are usually better suited to other
applications such as databases and Electronic Data Interchange (EDI) software. In the
author’s own experience, spreadsheet models have been developed which include
aspects as diverse as data entry and validation, file conversion, feeder file production,
data file merges, what-if analysis and data targeting. Whilst many of these aspects
can, and it could be argued should, be carried out by databases or specialist file
manipulation software such as Amtrix, the fact remains that the functionality is
available in spreadsheets and can work successfully as an integrated system alongside
many of the more traditional spreadsheet functions such as number crunching.
The term spreadsheet was originally an accounting term for a large sheet of paper split
into columns and rows. In 1978 a student at the Harvard Business School, Dan
Bricklin, and an employee of Interactive Data Corporation, Bob Frankston, coinvented and co-created a software program called VisiCalc. This is generally
accepted as being the first ‘electronic spreadsheet’. [Power, Bricklin]. The roots of
this program are still visible in the now ubiquitous Excel spreadsheet. [See below]
8
In 1979 Bricklin and Frankston formed Software Arts Corporation and, in conjunction
with a company called Personal Software, began marketing VisiCalc. This version of
VisiCalc ran on an Apple microcomputer and sold about 1 million copies during its
lifetime.
In the early 1980s IBM launched a PC based upon an Intel processor. At the same
time legal problems developed between VisiCorp (originally Personal Software) and
Software Arts. As a result VisiCalc lost ground on a new spreadsheet developed by
Mitch Kapor, known as 1-2-3. Mitch Kapor was a former employee of Personal
Software who initially offered his employers his spreadsheet. Personal Software
declined, and Kapor went on to found Lotus Development to produce his spreadsheet.
Lotus 1-2-3 quickly established itself as the industry standard spreadsheet. In 1985
Lotus took over Software Arts and consequently the VisiCalc spreadsheet. VisiCalc
was then abandoned on the grounds that it lacked functionality compared to Lotus 12-3 and Lotus Symphony.
At around the same time, a company called Microsoft developed a spreadsheet called
Excel for the 512K Apple Macintosh computer. This was one of the first spreadsheet
programs to utilise a graphical user interface (GUI) and to use a mouse as an input
device. The GUI was considered more user friendly and as a result many people
bought Apple Macintosh computers primarily to use Excel. A later incarnation of
Excel was one of the first applications to run on the Microsoft Windows operating
system. The lead established by Microsoft in using Windows was instrumental in
allowing MS Excel to become the definitive spreadsheet in industry today.
9
Academic Research
Why Worry About Spreadsheet Errors?
Although spreadsheets are now widely used in business, errors in these spreadsheets
only become an issue when the information produced by the spreadsheet is used. If,
for example, a spreadsheet is used to calculate the payback period for two mutually
exclusive investments, and returns three years for the first, and five years for the
second, even though the first project calculation should result in three years and three
months, it is unlikely that the error will have any impact. Unfortunately, this is not
always the case as the incidents outlined below illustrate:
Spreadsheets can be hazardous to your health [Ditlea 1987]
A company used an @sum formula to total a range of cells. When an item was added
to the list, the range was not changed. This led to the new data being excluded from
the calculation and the company underbidding for a project by $1/4m.
Re: ’Computer Error’ in Durham N.C. Election Results [Woodburg 1989]
In North Carolina, election results were calculated on a spreadsheet. The results were
incorrect and about to be posted when Woodburg, using a calculator, spotted the error.
Computer error costs graduate job [The Daily Telegraph 23/7/98]
In 1998, ten days after receiving a 2:1 in her degree, a student heard that the results,
calculated on a spreadsheet, were incorrect and she should have been awarded a 2:2.
This caused a job offer she had received to be withdrawn.
How Personal Computers Can Trip Up Executives [Business Week 24/9/84]
Two 15,000-cell spreadsheets were used to predict the market for CAD equipment.
Within these spreadsheets was a cell containing an inflation factor of 6% (1.06).
Unfortunately all the cells in the spreadsheet were rounded down using the built in
rounding down function, meaning that the inflation factor become 1 and thus inflation
was not added to the projections. This resulted in the market being underestimated by
$36m. It should be noted here that had only the format of the cell been rounded down
rather than the rounding down function being used, the calculation would still have
been done correctly.
10
Netfuture on-line newsletter (1997)
Fidelity, an investment firm, announced a $4.32 per share capital gains distribution
for shareholders in the Magellan fund. However, a month later the forecasted gain of
$2.3bn that gave rise to the dividend was changed to a $0.1bn loss. It transpired that a
clerical worker, whilst entering a $1.2bn spreadsheet ledger entry, had entered the
wrong sign, causing the miscalculation.
The cases above have all been documented because they have had a significant impact
on the situation in which the spreadsheet was involved. In the paper ‘How Important
are Spreadsheets to Organisations’ [Cleary 2000], Cleary, whilst acknowledging the
amount of research into spreadsheet errors and the conclusions reached that
spreadsheet errors are commonplace, argues that all this research could be classed as
irrelevant if spreadsheets are not actually having any impact on business decisions.
Cleary intends to attempt to quantify the importance of spreadsheets in business
decision-making.
How Frequently Do Spreadsheet Errors Occur?
As a result of the published impact of spreadsheet errors such as those outlined above,
much academic research has been done in an attempt to quantify the problem of
spreadsheet errors.
Hitting the Wall: Errors in Developing and De-bugging a ‘Simple’ Spreadsheet
Model [Panko 1996]
Panko observed 72 undergraduate MIS Business students develop a simple
spreadsheet model based on the cost of building a wall. The specification amounted to
10 sentences and was designed to minimize the need for domain knowledge. The
model answer had 25 non-label cells.
Panko found that 27 of the 72 spreadsheets developed (38%) had errors, however only
2 of the models had 2 errors, the remainder having just one, giving rise to a cell error
rate (CER) of 1.7%. He also found that most errors occurred only once, suggesting an
almost random occurrence of error making. It was also the case that 57% of the errors
detected were omission errors, whist the remainder were logic errors.
Panko then asked those subjects whose spreadsheets had errors to attempt to debug
their own spreadsheets. Only 16% (3 of 19) corrected their models. This, he
concluded, suggests that spreadsheet auditing is very difficult, particularly when
auditing ones own spreadsheets.
11
Panko concludes that, as these error rates in development and debugging were
consistent with traditional programming, spreadsheet development should be treated
in the same way. As Panko says, the process of adopting programming techniques to
spreadsheet development will be ‘like teaching a new dog old tricks’
The Subversive Spreadsheet [Butler 1997]
Butler warns of the danger in not taking the inherent risks in spreadsheets seriously.
He reports that in 1992 UK Tax inspectors computer audits found that 10% of
spreadsheets audited containe d material errors in relation to VAT calculation. Butler
then acknowledges that the extent to which spreadsheets need to be audited can be
identified using risk analysis based upon the complexity, number of users, size,
decisions based upon the model, level of users within the organization, and regulatory
issues. He subsequently advocates the use of computer assisted auditing tools such as
The Spreadsheet Detective (although the UK Tax Inspectors now use a custom built
package called SpACE).
What We Know About Spreadsheet Errors [Panko 1998]
Panko concluded that error rates in Spreadsheets are “in line with those in programming and
other human cognitive domains”. This paper summarised the Spreadsheet errors research
that had taken place. He found that since 1997, 91% of 54 Spreadsheets contained
errors, and since the study began in 1987, 24% contained errors [see table below].
Studies of Spreadsheet Errors
Author(s)
Cell Error
Rate
(CER)
Pct. of
Models
with
Errors
Sample
Study
19
operational
spreadsheets
14 had qualitative
unspecified.
273
operational
spreadsheets audited
by
143
United
Kingdom
tax
inspectors in 1992.
20
operational
spreadsheets from 10
firms.
Only counted "material" errors. Inspectors used
a spreadsheet analysis program designed to
identify suspicious parts of a model. Such
programs identify only some errors.
Hicks [1995]
1 module with 19
submodules about to
enter operation.
Part of a capital budgeting system for NYNEX.
Checked heavily ahead of time. Code
inspection by three analysts. Found 45 errors in
3,856 cells.
Coopers
&
Lybrand [1997]
23 spreadsheets from
industry
Spreadsheets off by at least 5%
91%
KPMG [1997]
22 spreadsheets from
industry
Spreadsheets containing major errors.
91%
Lukasic [1998]
Butler [2000]
2 spreadsheets
7 spreadsheets for
tax submissions
Field Audits
Davies
[1987]
&
Ikin
Butler [1992]
Cragg
[1993]
&
King
errors.
Methodology
21%
10.7%
150 to 10,000 cells. Had been in use for
median of 6 months
25%
1.2%
2.2%, 2.5%
Spreadsheets
audited
with
enhanced
methodology and software. Single auditor.
12
100% (1)
100 %
86%
Cell Error
Rate
(CER)
Pct. of
Models
with
Errors
Author(s)
Sample
Study
Total
367 spreadsheets
Weighted average
24%
Since 1997
54 spreadsheets
Weighted average
91%
Cell
Entry
Experiments
Olson & Nilsen
[1987-1988]
Errors counted when made, even if corrected
later
14 experienced Lotus
1-2-3 users.
Floyd & Pyun
[1987]
Lerch [1988]
21 professionals with
at least one year of
Lotus
1-2-3
experience.
Development
Experiments
Filled in formulas in skeleton model. 4 formula
cells per person. 56 total.
21% (2)
Reanalysed Olson & Nilsen [1985, 1987-1988]
for errors in text cells.
12.5%
Filled in formulas in template. CER based on
formulas only. CER especially high for formulas
referring to cells in both different rows and
different columns.
Errors counted at end of development process
Brown & Gould
[1987]
9 highly experienced
spreadsheet
developers
Brown & Gould
[1987]
Developed 3 models apiece. All made an error
in at least one model. Minimum definition of
errors, excluding the omission of requirements.
Broader definition of errors including the
omission of requirements.
Hassinen [1988]
92
novice
spreadsheet students
developed
355
spreadsheets
Paper and pencil exercise. Subjects filled in
formulas in a skeleton model containing
organization and numbers.
Hassinen [1988]
10 novice students
developing
48
spreadsheets
61 upper division
business
and
graduate accounting
students
Computer exercise for same task.
Janvrin
&
Morrison [1996,
2000]
88
senior-level
accounting students
Panko
Halverson
[1997]
&
35
undergraduate
business
students
working alone
Study 2: Developed model with multiple
worksheets. 66 links between worksheets. No
templates to work from; only 1 check figure.
CER is percent of incorrect links between
spreadsheets.
Developed pro forma income statement based
on the Galumpke task.
Panko
Halverson
[1997]
&
40
undergraduate
business
students
working in groups of
4
Developed pro forma income statement based
on the Galumpke task
Panko
&
Sprague [1999]
102 undergraduate
business
students
and
50
MBA
students. 17 MBAs
had more than 250
hours of experience
Developed a model based on the Wall task
designed to be relatively si mple and free of
domain knowledge. No difference in errors
across groups.
Teo
&
[1997]
168 undergraduate
business
students
taking second-year IS
course
Developed a model based on the Wall task,
then did a what -if analysis
80
undergraduate
business students
Developed spreadsheet working alone or in
triad (group of 3). Error rates for individual,
triad.
Janvrin
&
Morrison [1996,
2000]
Tan
author
(unpublished)
Code Inspection
Experiments
Galletta
[1993]
et
11.3% (2)
al.
Study 1: Developed model with multiple
worksheets. Had template with filled -in values.
Measured incorrect links between worksheets.
Model had 51 links. Students had 16 days to do
task. Worked an average of 8.8 hours
44%
63%
4.3% (2)
48%
7%-14%
(3)
Examined 6 small model with 1 seeded error in
each. Subjects with 250 hours or more of
13
84%-95%
8%-17%
(3)
5.4%
80%
2.0%
60%
2.0%
35%
2.0%
42%
4.6%, 1.0%
86%, 27%
Subjects looked for errors in a model
30 MBA students and
30 CPA accountants
55%
34%-54%
Author(s)
Sample
Cell Error
Rate
(CER)
Study
spreadsheet experience did not find more
errors than those without experience.
Galletta
[1997]
et
al.
113 MBA students
Finding eight seeded errors in a student
budgeting model.
Panko
and
Sprague [1998]
23
undergraduate
subjects with errors in
initial model.
Study described above. Code inspected own
models. Fixed 18% of errors. Only 13% of
spreadsheets were fixed completely. One was
made worse.
Panko [1999]
33
undergraduate
MIS majors
Code inspected 11 spreadsheet models,
individually and then in groups. Missed errors
for individuals, groups.
Pct. of
Models
with
Errors
(4)
45%-55%
(4)
81% (4)
40%, 17%
(4)
Notes:
(1) Errors per model and percent of models with errors computed on basis of submodules.
(2) Errors per formula cell.
(3) Errors per inter-spreadsheet link
(4) Percent of seeded errors not detected
This increase reflected the increased complexity of spreadsheet models and the
improvement in auditing methods. Panko also argued that statistics such as the
percentage of spreadsheets containing errors and average number of errors in a
spreadsheet were flawed, and a more informative statistic would be the Cell Error
Rate (CER) which is the percentage of errors as a proportion of numerical and
formulae cells (i.e. labels are excluded from the calculation). This is similar to the
errors per ELOC (effective lines of code) metric used in traditional programming
disciplines. Panko went on to suggest that errors could occur at various stages of the
life cycle of the Spreadsheet. He suggested that these errors could occur at any of the
following stages:
•
•
•
•
Errors during cell entry – often corrected at the time
Errors at the end of the de velopment stage (pre-audit)
Errors found and missed in code inspection
Errors in operational Spreadsheets
Panko concluded that error detection is not exact and that because errors occur in a
certain percentage of spreadsheets does not mean the others are error free, i.e. they
may have undetected errors.
A Taxonomy of Error Types [Panko & Halverson 1996]
This paper proposed a classification system for different types of errors. Panko and
Halverson suggested that errors could be divided into two groups, quantitative and
qualitative errors. They then suggested that the former group could be sub-divided
into three sub-groups, mechanical errors, logic errors and omission errors. The
quantitative errors were defined as those that had an immediate impact on the
accuracy of the spreadsheet, whereas qualitative errors were those that, whilst having
no immediate impact on the accuracy of a spreadsheet, may cause inaccuracies in the
future. An example of a qualitative error would be overwriting a formula with a valu e
14
returned. Whilst this would be accurate as long as the contributing cells remained
unchanged, when one of the contributing cells was changed, the formula result would
not reflect this and the end result would be incorrect. Panko and Halverson defined a
mechanical error as one that was caused by mistyping a value, or erroneously entering
a formula by pointing to the wrong cell. An example of a logic error could be entering
an incorrect formula because the user/developer did not totally understand the
situation. An omission error, as the name suggests, could caused by the user missing a
vital piece of information when creating the spreadsheet. In addition to proposing a
taxonomy of error types, in this paper Panko and Halverson also concluded the
following:
•
•
•
•
All research done on spreadsheet errors found an unacceptable amount of errors.
Apparent simplicity and ease of use in spreadsheets leads to overconfidence on the
part of the user and subsequently leads to errors.
Spreadsheet auditing is regularly either not done, or is done to a very basic
standard. Code inspection in spreadsheets is very rare.
The apparent simplicity of spreadsheets has led to end users producing their own
spreadsheets and, in the process, control and development methods and policies
are non-existent or overlooked.
15
Classification of Spreadsheet Errors [Rajalingham et al 2000]
Here the authors describe a framework for the classification of spreadsheet errors. The
framework was defined as a result of an investigation of the nature, source , types and
impact of errors documented in previous spreadsheet research. This framework is
summarised in the table below:
Taxonomy of Spreadsheet Errors
SPREADSHEET ERRORS
•
SYSTEM-GENERATED
• USER-GENERATED
• QUANTITIVE
• ACCIDENTAL
• DEVELOPER (workings)
• Omission
• Alteration
• Duplication
• END-USER
• DATA INPUTTER (input)
• Omission
• Alteration
• Duplication
• INTERPRETER (output)
• Omission
• Alteration
• Duplication
• REASONING
• DOMAIN KNOWLEDGE
• READ-WORLD KNOWLEDGE
• MATHEMATICAL
REPRESENTATION
• IMPLEMENTATION
• SYNTAX
• LOGIC
• QUALITATIVE
• SEMANTIC
• STRUCTURAL
• TEMPORAL
• MAINTAINABILITY
16
Other than the distinction between system generated and user generated made at the
top level, this classification largely concurs and expands upon that put forward by
Panko and Halverson [Panko & Halverson 2000] in aforementioned paper ‘Taxonomy
of Error Types’, in that both papers make a high level distinction between quantitative
and qualitative errors and split these into further sub-categories. In the case of
Rajalingham et al, the source of the error is given a greater significance in the
classification, and this comes back into the classification with the distinction between
developer and end user, within the accidental category. This classification also splits
the qualitative category into semantic and maintainability, a distinction not explicit,
but certainly possible, with Panko and Halverson’s classification.
A Survey of Research on Spreadsheet Risks [Panko & Halverson 1996]
This paper suggested a taxonomy for understanding issues in spreadsheet risks, a
spreadsheet research risks cube. This suggested that there are three dimensions to
consider when understanding spreadsheet risks. These are dependent variables, stages
in the spreadsheet’s life cycle and the methodology used to create the spreadsheet.
The authors then go on to investigate research into the first two of these dimensions,
arguing that the third dimension, methodology, had already been adequately covered
in previous academic research.
Whilst discussing dependent variables, the authors expand upon their own taxonomy
of error types by suggesting that logic errors can be split into ‘Eureka’ errors (those
that are obvious as soon as they are spotted), ‘Cassandra’ errors (those that are
difficult to prove, even when spotted), pure logic errors (errors down to an obvious
logical mistake) and domain logic errors (those that are down to a misunderstanding
of the real-life situation in which the model will be used). The authors go on to
discuss errors in different stages of the life cycle of the spreadsheet, and at each stage
found that errors are common and, in contrast to a traditional programming discipline,
control and procedures are rare and un-observed. The authors concluded that similar
rules and procedures to those used in traditional programming should be used in the
development of spreadsheets.
Two Corpuses of Spreadsheet Errors [Panko 2000]
Panko also conducted a series of experiments designed to assist the assessment of
spreadsheets for errors. To do this he put forward two corpuses of errors seen in
spreadsheet experiments. By using two separate tasks, the Galumpke Corpus and the
Wall Corpus, he tested spreadsheets develope d by students in laboratory experiments.
He documented the results reporting quantitative errors only, split into the three types,
mechanical, logical and omission. He found that using these corpuses, errors were
diverse, and the ratio of tokens (individua l errors) to unique errors was very low. He
concluded that any error reduction tools must not simply detect one particular type of
error, but rather encompass a wide range of errors in order to be effective/useful. He
also suggested that the two corpuses used within the experiments should be useful to
other spreadsheet researchers in testing for errors. As a broad conclusion Panko
17
decided that research into the spreadsheet error field shared many similarities with
research into programming errors and human cognitive errors. Overall, Panko
concluded that the research done and the literature available suggested that only cellby-cell team code inspection is likely to significantly reduce errors, and even that by
only 80%, a similar conclusion to that drawn in most of the software development
auditing research.
In addition to the user/developer -generated errors, which have attracted the most
research, it must be noted that some errors occur beyond the influence of the user or
spreadsheet developer. Both Lotus and Microsoft have acknowledged that rounding
errors can occur in 1-2-3 and Excel respectively. According to Microsoft, Excel will
sometimes produce a rounding error in the 14th or 15th decimal place [Computing
Magazine 2/3/95]. In addition, the computer it self can cause mathematical error, for
example, when Intel launched the Pentium processor, it was found to suffer similar
rounding errors.
What Can Be Done To Avoid Spreadsheet Errors?
Whilst the apparent simplicity of spreadsheets has been a major factor in their
popularity, it is clear that spreadsheets, despite appearances, are anything but
simplistic. Research has repeatedly shown that development of all but the most basic
Spreadsheets should be considered as complex as developing a computer program in a
traditional programming language.
The Institute of Chartered Accountants in Ireland [ICAI Factsheet 1] argue that
spreadsheet development is similar to traditional programming in that both fields
involve the following 3 key elements:
•
•
•
A business problem is identified
A solution is constructed using computer based tools
The solution is implemented by the business to address the problem
An Approach to the Teaching of Spreadsheets using Software Engineering
Concepts [Chadwick et al 1999]
In this paper the authors identified two generic types of skills needed to develop both
spreadsheet models and traditional software, enabling skills and planning skills.
Enabling skills can be further split into generic enabling skills (such as the ability to
use spreadsheets and understand spreadsheet concepts) and specific enabling skills
(those specific to a particular spreadsheet program or programming language). The
planning skills can also be divided into generic planning skills (those achieved by
understanding the problem situation or business domain) and specific planning skills
(such as those relating to data modelling and best practice development techniques).
18
As a result of these similarities the authors changed the way that spreadsheets were
taught to students to one that was largely software engineering based in its approach.
The authors recommend the use of modularisation and a stepped life cycle approach
to spreadsheet development. The modularisation approach involves identifying
individual modules within a business situation, and the links between these modules,
and designing and developing spreadsheets based on theses individual modules. The
life cycle approach taught was given the acronym RADAR (Requirements, Analysis,
Design, Acceptance, Review).
These approaches were tested as a new method of teaching spreadsheets to students in
1998. The results showed an increase in the average marks in all five disciplines
taught, suggesting that the software engineering approach utilized was useful.
What Constitutes Good Spreadsheet Design?
As has been shown, spreadsheets are incredibly prone to errors, and there are many
similarities between spreadsheets and software engineering. Many studies have found
that one way to reduce the occurrence of errors in spreads heets would be by
improving the way in which spreadsheet models are designed. As a result many
people have published guidelines or methodologies aimed at reducing the errors in
future spreadsheet models.
The Subversive Spreadsheet [Butler 1997]
Following on from the points made earlier in this report, Butler, in the same paper,
lists what he considers to be good practice in spreadsheet design. He includes the
following factors as guidelines in the design of spreadsheet models:
•
•
•
•
•
•
•
•
•
•
Spreadsheet specified and planned on paper
Spreadsheet documented on the first worksheet
Documentation within the spreadsheet to include purpose of spreadsheet,
developer, issue dates, a brief summary of contents, a statement of assumptions,
instructions for use, an explanation of key formulae, and features
All constants should be stored in easily identified named ranges
Complex formulae should be broken down into many simple formulae
Spreadsheet tested with ‘silly’ numbers
Spreadsheet reviewed by someone other than the developer
All relevant cells should be protected
Password protected
Spreadsheet subjected to post implementation audit(s)
19
Spreadsheets [Binning 1999]
Binning puts forward the following as steps in developing a good spreadsheet:
1. Determine the objective of the spreadsheet
2. Analyse the potential end-user applications – Input/Output/Transformation of
Data/Data Storage/Control
3. Design the spreadsheet using either a modular or hierarchical design
4. Test the spreadsheet application
5. Document the spreadsheet application
Modelling Optimisation Problems In The Unstructured World Of Spreadsheets
[Ragsdale & Conway 1997]
This paper argues that the four goals of spreadsheet design are communication,
reliability, auditability and modifiability. As a result they put forward the follow ing 8
spreadsheet design guidelines:
1.
2.
3.
4.
5.
6.
Do not embed numeric constants in formulae
Organise the data then build the model around the data
Related items should be grouped together (modular approach)
Try and use formula that can be copied where possible
Columns and rows should be totalled at the end of the column/row
Remember that in English, the eye moves from left to right so arrange the
spreadsheet to follow this convention.
7. Use shading, borders and protection to identify changeable cells
8. Use text boxes to document the model and cells within the model
Quality Control in Spreadsheets: A Software Engineering-Based Approach to
Spreadsheet Development [Rajalingham et al 2000]
This paper argues that as it is widely accepted in spreadsheet research that a
sprea dsheet model is, in effect, a computer program, the development of spreadsheets
should draw largely on a software engineering approach. The authors therefore put
forward the following approaches to spreadsheet development:
1. Analyse the situation thoroughly , identifying all data and operations necessary
to resolve the problem
2. Document all formula in a hierarchical tree form based on the software
engineering technique of hierarchical decomposition.
3. Separate the components of a spreadsheet into data and operations
components.
4. Where necessary to increase readability and understanding of a spreadsheet,
use copies of data component cells in the operations components
5. Protect the operations components of the spreadsheet
20
A Structured Methodology for Spreadsheet Modelling [Knight et al 2000]
The authors of this paper recommend the use of a structured software development
methodology based on Jackson diagrams to analyse problem situations. They
recommend Jackson diagrams, as they are useful in the modularisation and the
application of a clear structure to spreadsheet models. They go on to advocate the
separation of data input, processing and output modules of a spreadsheet model.
Efficient Methods for Checking Integrity: An Integrated Spreadsheet
Engineering Methodolo gy (ISEM) [Rajalingham et al 1999)
This paper concludes the author’s search for a spreadsheet development methodology
investigated in the aforementioned papers. Here the authors have developed a
spreadsheet methodology, the Integrated Spreadsheet Engineering Methodology
(ISEM). At its core, ISEM is based upon the traditional systems development life
cycle used extensively in software engineering. ISEM basically involves following
the steps and stages outlined below:
•
Stage 1 – Planning
•
•
•
•
Stage 2 – Analysis
•
•
•
•
Step 1: Redefine/define the spreadsheet problem – This involves identifying
the scope of the system along with the data sources/destinations and the
transformation of the data required.
Step 2: Understanding the existing system (if one exists)
Step 3: User requirements and constraints of the spreadsheet system
Stage 3 – Design
•
•
•
Step 1: Request for construction of the spreadsheet system – usually instigated
by the management or owners of a business problem
Step 2: Initial investigation of the spreadsheet system – this step incorporates a
top level design of the current and/or proposed system
Step 3: Feasibility study of the spreadsheet system – this step is essential to
decide whether a spreadsheet is the correct platform to use for the proposed
system.
Step 1: Logical Design – a high level unambiguous representation of the
proposed system
Step 2: Physical Design – a low level design of the spreadsheet model
identifying specific functions and the locations of each module within the
spreadsheet model
Stage 4 – Implementation
21
•
•
•
•
•
•
Step 1: Spreadsheet system building – the actual construction of the
spreadsheet
Step 2: Testing of the spreadsheet system – including a test of the spreadsheet
against the design and a test of the functionality by feeding test data through
the spreadsheet.
Step 3: Document the spreadsheet system/program – within the spreadsheet
and/or on paper
Step 4: Operation of spreadsheet system
Step 5: Post-implementation review of the spreadsheet system
Stage 5 – Maintenance
The authors recommend various tools and techniques such as tree-based formula
analysis, formation of data matrices and the use of software auditing tools to assist in
the development and the application of the above steps.
Spreadsheet Modelling Best Practice [Reed & Batson 1 999]
Reed and Batson endorse the software engineering approach to spreadsheet
development. They suggest that spreadsheet development should follow a modelling
life cycle similar to those in use in software engineering as shown below:
The Six Stages Of The Modelling Life Cycle
Reed and Batson acknowledge that spreadsheets can vary from the very simple to the
very complicated and therefore put forward four generic model types to determine
22
how much of each stage of the modelling life cycle should be applied to the
development in question. The four generic types are:
•
•
•
•
Complex models with adequate time for development
Simple models with low risk
Time critical models
Ill-defined problem models
Reed and Batson’s best practice guide then goes on to endorse and in tegrate many
tools and techniques such as input/process/output module separation, bubble
diagrams, prototyping, documentation, simplification of formulae, test plans, test
documentation, version control, and others in use in software engineering and
spreadsheet development approaches.
How Can We Audit Spreadsheets?
As can be seen, even since the introduction of best practice guidelines and
methodologies, spreadsheet errors still exist. It is therefore essential that spreadsheet
models are correctly audited. Many studies have been done to try to identify the
success rates of different ways of auditing spreadsheets.
Risk Assessment for Spreadsheet Developments: Choosing which Models to
Audit [Butler 2000]
Butler argues that to efficiently use resources, intensive auditing such as group code
inspection should only be carried out on selected spreadsheets. To illustrate this
Butler describes the process used in H.M. Customs and Excise to assess spreadsheets.
To determine which spreadsheets qualify for intensive auditing, the auditor needs to
establish both the likely incidence of errors in the model and the impact on the
organisation of those errors. If both these aspects are deemed significant, then an
identification of the scope and resources required, coupled with a more detailed risk
analysis takes place. Once this has been completed the critical question, “Does the
risk outweigh the resources required?” must be asked. If the answer is yes, full code
inspection will take place. In H.M. Customs and Excise, their auditing software,
SpACE, assists this code inspection.
Butler concludes that despite research showing that poor practice in spreadsheet use is
rife, and good practice in spreadsheet development is rare, users have an un-supported
confidence in the data produced and regularly use them for major business decisions.
Butler also notes that as in any other system, the accuracy of any results is at the
mercy of the data entered in to the system and that a correct model does not always
equal correct output.
23
Integrity Control of Spreadsheets: Organisation and Tools [Rajalingham &
Chadwick 1998]
Rajalingham and Chadwick advocate checking a spreadsheet throughout its lifetime
using Di Antonio’s Method for Spreadsheet Development which provides 6 steps for
the construction of a spreadsheet, Chadwick et al’s [Chadwick et al 1997] 5 step
methodology for auditing during design and production, which involves checking
individual cells and modules and then a software auditing tool, The Spreadsheet
Professional, for post development auditing. They argue that a modular approach to
auditing, already well established in software engineering, should be applied to
spreadsheet auditing. This approach involves identifying constituent modules within a
spreadsheet. These modules are usually in clear blocks (on a reasonably well
developed spreadsheet) and are largely self -contained, although they can have
dependent or precedent links to other modules (known as coupling). Once these are
identified the cells in these modules are labelled on a row and column basis by adding
labels to the spreadsheet where necessary. These labels are then used to provide either
algebraic English, full English or graphic display descriptions to the formulae for
more intuitive auditing. A survey carrie d out by the authors found that 73% of
respondents preferred the visual methods for auditing, with the graphic display
method proving the most popular method of displaying the formulae for auditing.
Visual Checking of Spreadsheets [Chen & Chan 2000a]
In this paper the authors make the observation that “Spreadsheets are easy to use and very
hard to check”. They argue that a major reason for this is the difference between the
structure of a spreadsheet that a user sees, known as the surface structure, and the
actual structure of the spreadsheet, known as the deep structure. They argue that the
more different these structures, the more difficult the spreadsheet is to understand.
The surface structure will typically be defined by the layout of the cells, the empty
cells and headings/labels applied to the spreadsheet. The deep structure is, of course,
the way the computer sees the data, as a series of interdependent cells. Chen and Chan
then illustrate the different ways n which auditing software can bridge the gap
between the surface and the deep structure by visually identifying types of cells,
formula schema and cell interdependencies. This drawing together of the two
structures is the primary aim of all good software auditing tools.
In a further paper [Chen & Chan 2000b] the authors concluded, “users prefer visual
interactive tools rather than paper printouts or screen scanning”
24
Quality Control in Spreadsheets: A Visual Approach using Colour Coding To
Reduce Errors in Formulae [Chadwick et al 2000]
Chadwick et al (2000) proposed two methods of enhancing spreadsheets to ease
auditing. The first dealt with a visual approach to displaying formulae, whilst the
second looked at identifying original formulae and where these had been copied.
Four methods of displaying formulae to the auditor were proposed, the algebraic
method used by Excel, a method that applies labels to replace the cell addresses whilst
retaining the Excel formula, an almost complete English method, and a visual display
method which indicated, using colour and column headings, the result of the formulae
and applied colour coding to the columns and rows involved. A survey amongst 63
students was then done to determine which method was preferred. The visual method
was the most popular with 34% of the students, with the algebraic Excel method
coming second with 26%. The complete English and applied labels methods achieved
22% and 18% respectively.
The second problem, arising from copying formulae, involves a combination of
colour coding origin al and copied formulae, whilst adding a comment to the original
cell. A small survey of five students came out firmly in favour of this method,
particularly as this left the numeric and label cells as black and thus errors caused by
overwriting formulae were easy to spot.
Chadwick et al conclude that the visual approaches to spreadsheet auditing were
proven to be useful and advocated the development of tools to enable the automation
of these visual features.
Applying Code Inspection to Spreadsheet Testing [Panko 1999]
Panko observed 60 undergraduate MIS students doing a code-inspection of a
spreadsheet containing seeded errors. They inspected the spreadsheet firstly on an
individual basis, and subsequently in groups of 3.
The results showed that the groups found 83% of the errors, compared with 63% on
an individual basis. However, further analysis of the results suggests that no new
errors were found using groups, rather that the results were the same as could be
achieved by the amalgamation of the errors found in the individual inspections. This
suggests that the group inspection phase was perhaps not essential, providing the
capability exists to amalgamate the errors found on an individual basis for eradication.
Panko concluded that whilst the groups identified 83% of the errors, this still left 17%
of errors undetected, thus proving that whilst code inspection may be useful, it is not,
as Panko puts it, “the mythical silver bullet that will remove all errors ”
25
Are Two Heads Better Than One? (At Reducing Errors in Spreadsheet
Modelling) [Panko & Halverson 1996c]
This paper attempted to measure the different levels of success enjoyed by individuals
and groups of two and four in the design and audit of a pro-forma Income Statement.
The results showed that in a total of 99 models, 182 errors were made, which
accounted for 51 different errors. This suggested that there was an almost random
element for error generation. The results suggested that statistically, there is no clear
indication that groups of two are significantly better than individuals. However the
results did suggest that groups of four (tetrads) have more success than either groups
of two or individuals although it was noted that even the error rates produced by the
tetrads would be unacceptable in a ‘real life’ system. The study found that, rather
surprisingly, groups tended to spot and correct omission and ‘Eureka’ logic errors,
whilst they often overlooked mechanical errors as they were made by colleagues.
Spreadsheet Errors: What We Know. What We Think We Can Do [Panko
2000]
In the second part of this paper, Panko looks at what we can do to reduce spreadsheet
errors. Panko acknowledges that we cannot eliminate errors completely, but identifies
some methods of reducing errors. He advocate s cell protection and re-keying of data
for verification purposes as good methods for avoiding hardwiring and data input
errors. Panko, whilst acknowledging that good spreadsheet design is good practice,
stresses that this alone will not significantly reduce errors. He is much more
supportive of code inspection and test data testing as worthwhile and possibly the
most reliable methods of reducing errors. Panko also touches upon the use of error
discovery software and whilst agreeing that it could be useful, calls for some testing
of these products before they can be classed as safe and effective. The second part of
this report attempts to address this issue.
26
Trends in Academic Research into Spreadsheets
Early 1980s
Author
Davis GB (1982)
Creeth (1985)
Late 1980s
Author
Batson
(1986),
Williams (1987)
Brown & Gold
(1987), Davies &
Ikin (1987), Floyd
& Pym (1987),
Olsen & Neilson
(1987-1988),
Lerch
(1988),
Hassinen (1988)
Early 1990s
Author
Neilson
(1991),
Mackay & Lamb
(1991)
Nordi & Miller
(1991), Galletta &
Hufnagel (1992),
Cragg & King
(1993)
Late 1990s
Author
Butler
(1995),
Hicks
(1995),
Galletta, Hartzel,
Johnson & Joseph
(1996)
Floyd, Walls &
Marr (1995), Hall
(1996), Speier &
Brown (1996)
Conclusions/Contents
“There may be problems with end user development as they are
untrained and undisciplined”
The apparent ease of use is a major factor in the underestimation
of the scope for errors. Spreadsheets are not always the right
tool for a task. A third of spreadsheets have errors.
Conclusions/Contents
Began to develop design/testing/auditing methods and
methodologies
Performed many experiments on spreadsheet errors and
concluded that errors were rampant in spreadsheets.
Conclusions/Contents
Looked at the impact of spreadsheet errors and the need for
training of spreadsheet developers and end users.
Examined the use and effect of standard procedures and controls
in spreadsheets.
Conclusions/Contents
Investigated spreadsheet auditing methods and its success.
Examined the use, complexity and development controls in
spreadsheets.
Adapted from ‘Review of Errors in Spreadsheets’ [Creely 2000]
27
Academic Research Conclusions
Having reviewed the academic research to date, the following conclusions can be
drawn:
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
Despite much research into the causes of spreadsheet errors and ways of avoiding
spreadsheet errors, these errors are still the rule rather than the exception.
Spreadsheet errors do not cause problems, but acting upon results affected by
spreadsheet errors can.
Despite appearances to the contrary, spreadsheets are arguably the most powerful
application used at a development level by non-technical users.
Error rates in spreadsheet development and debugging are consistent with those in
traditional programming
Error rates in ‘live’ spreadsheets are so high because the steps followed in
traditional programming to reduce these errors are overlooked in spreadsheets.
This is largely due to the idea that spreadsheets are simple which is still prevalent.
Errors can be classified, based on types of error, source of error and/or time of
error.
Whilst cell-by-cell team code inspection proves the most successful method of
spreadsheet auditing, this still only produces an 80% success rate.
Software Engineering concepts can be applied successfully to spreadsheet model
development and auditing.
Many spreadsheet development methodologies/guidelines are based on Software
Engineering principles and concepts.
Good spreadsheet design is essential in reducing spreadsheet errors and increasing
readability and maintainability.
An efficient use of resources will usually require some form of risk analysis to
decide to what depth a spreadsheet model should be audited.
The conceptual difference between what a user sees and what the computer sees in
a spreadsheet is the main reason that auditing spreadsheets is so difficult.
Auditing is helped by using a visual approach to cell descriptions.
The visual auditing approach lends itself to the use of software auditing tools.
Spreadsheets are not simple!
28
Software Spreadsheet Auditing Tools
From the academic research performed relating to spreadsheet errors, it was
concluded that auditing of spreadsheets is incredibly difficult, particularly without the
use of a visual auditing approach. It was also found that even the best methods of
spreadsheet auditing only produce a success rate of around 80%. In order to address
these issues, software has been produced to assist in the audit of spreadsheets. These
software tools tend to provide a visual approach to assist the user in auditing the
spreadsheet. Most also tend to be aimed at the spreadsheet developer, often providing
additional functions to assist in the development of spreadsheets as well as providing
auditing functions. These tools range from those that merely assist in cell inspection
and audit, to those that attempt to identify unique formulae and potential problem
cells. To date, little, if any, research has been published that assesses the usefulness
and capabilities of these software tools. To address this issue, the second part of this
report will document an investigation into software auditing tools and attempt to
answer the following questions:
•
•
•
•
Are software auditing tools for spreadsheets useful?
How do software auditing tools assist the user in auditing spreadsheets?
What errors are software auditing tools good at identifying?
What errors are software auditing tools poor at identifying?
To answer these questions, four software auditing tools, along with the auditing
functions built in to Excel, were tested against a specially designed spreadsheet that
contained 17 seeded errors. The software tested was:
Excel’s Built-In Auditing Functions – These are included as standard functions in
Microsoft Excel, and for the purpose of the test were deemed to be primarily the
functions available on the Excel auditing toolbar.
The Spreadsheet Detective – A commercially available product that provides
extensive auditing functions.
The Excel Auditor – A commercially available product that provides functions for
both auditing and development of spreadsheets.
The Operis Analysis Kit – A commercially available product that provides auditing
and development functions for spreadsheets.
Spreadsheet Auditing for Customs and Excise (SpACE) – A tool in use at Customs
and Excise to allow auditors to audit a businesses VAT calculation spreadsheets.
29
The Sample Errors Spreadsheet
In order to investigate different spreadsheet auditing software, a Sample Errors
Spreadsheet (see appendix B) was developed. This spreadsheet is loosely based upon
a spreadsheet in use at the CWS that is used to produce flash turnover reports in the
case of an OLAS system failure (OLAS is the accounting software used by CWS
Retail). The system is used by taking a downloaded file, which is produced every
week from the OLAS system, from the previous week and manually entering this
week’s flash turnover figures. It should be noted that the spreadsheet has been
modified almost beyond recognition to allow the inclusion of various different types
of errors, and these errors ha ve been deliberately added to the spreadsheet to allow the
auditing software to look for them. It should also be noted that the figures in the
spreadsheet are not actual figures (to maintain CWS data confidentiality) and that the
spreadsheet is but one of a suite of six that provide the flash turnover reports in the
case of OLAS failure. This suite has, to date, only been used twice as it is purely a
back-up tool designed to replicate reports that are directly available from OLAS when
the system is available. The spreadsheet is split into three worksheets. The first of
these is used to enter flash turnover data for each of the categories identified for each
of the regions within CWS. The second sheet is used to hold the data downloaded the
previous week from OLAS and includes the rest of the data needed to produce the
report held on the third worksheet. Once the data has been manually entered into the
first worksheet, and the file loaded into the second worksheet, the report can be
produced and distributed.
The following shows the errors added to the spreadsheet:
Qualitative Errors (Q)
Formulas overwritten with values
Error
Number
1
2
3
4
Description
Cell INPUT!D27 was a formula and is now a value
Cell REPORT!D27 was a formula and is now a value
Cell REPORT!E13 was a formula and is now a value
Cell REPORT!J60 was a formula and is now a value
Fixed value used when a named cell reference should be used
Error
Number
5
Description
All of the REPORT!K column have a fixed value in the calculation rather
than referring to the cell named “INFLATION” on the INPUT worksheet
30
Presentation illogical
Error
Number
6
Description
On the report, Northern Ireland’s Non Food categories are switched over
compared with the others
Incorrect Protection
Error
Number
7
Description
On the input column of the INPUT sheet, the totals cells are left
unprotected. On the REPORT sheet, the whole of the ‘This Week’ column
is unprotected, which could lead to an untrained user entering the figures in
to the wrong cells.
Quantitative Errors – Mechanical (M)
Incorrect summation
Error
Number
8
9
10
Description
REPORT!E52 excludes row 51
REPORT!O59 includes O32 rather than the correct O34
INPUT!D33 excludes row 32
Quantitative Errors – Logical (L)
A formula copied down a column looks at the wrong column
Error
Number
11
12
Description
All of the REPORT!P column uses ‘Revalued Last Year’ (column K) rather
than ‘Year To Date’ (column H)
All of the REPORT!K column brings in ‘This Year To Date’ values from
the download file (which excludes this weeks values) and re-evaluates that,
rather than taking the value from the report which includes this weeks
values.
31
Incorrect Summation of a column including subtotals
Error
Number
13
Description
The formula in REPORT!E61 tota ls the whole column, rather than just the
subtotals or the actual detail lines.
Percentages added rather than calculated on a total cell
Error
Number
14
Description
Cell REPORT!P44 has a total of percentages rather than using the same
calculation as the detail lines to produce a percentage of the totals.
Percentages averaged rather than calculated on a total cell
Error
Number
15
Description
Cell REPORT!N52 has an average of percentages rather than using the
same calculation as the detail lines to pr oduce a percentage of the totals.
This can be more difficult to spot than a total of percentages and can at first
glance appear correct.
Quantitative Errors – Omission (O)
A row missing from all the worksheets
Error
Number
16
Description
All of South Midlands Non Food New rows have been deleted.
A row missing from one of the worksheets although it exists on the others
Error
Number
17
Description
The Southern Non Food New row has been deleted from the REPORT
worksheet
32
Method Used to Test Software Tools
In order to maintain a level of consistency across tests of spreadsheet tools, where
possible, certain guidelines were followed. The tests each followed a three-stage
procedure. The first stage involved the author becoming familiar with the software by
examining it in an open session in order to gain a thorough understanding of the
capabilities and limitations of the software, the way the software worked and to whom
the software was intended for use. This first stage allowed the author to inves tigate
functions included in the software tool that were beyond the scope of the test errors in
the Sample Errors Spreadsheet, to study any documentation provided with the
software tool, and answer the questions asked on the first page of the Spreadsheet
Auditing Software Test Form. The second stage involved the testing of the software
tool against the Sample Errors Spreadsheet and the completion of the Test Results
section of the Spreadsheet Auditing Software Test Form, including the allocation of a
result to each error tested based upon the following criteria:
PASSED
ALMOSTPASSED
ALMOSTFAILED
FAILED
The software spotted the error with minimal intervention from the
user, often highlighting the problem before the user examined the
erroneous cell(s) on an individual basis. The software made the error
much easier to spot by using visual display methods.
The software hinted strongly at the possibility of the error and after a
little investigation, the user, aided by the display tools in the software,
was able to find the error.
The software suggested the presence of an error but failed to identify
the rogue cell(s). An investigation by the user found the offending
error.
The software failed to provide any suggestion that the error existed.
Criteria for awarding test results
The second stage tests were, with the exception of the Excel built in auditing
functions test, completed by allowing the software to guide the author to identification
of each error. The Excel built in auditing functions were, due to the limitations in the
way they work and the fact that they are available for use alongside the other tools,
were tested on a cell-by-cell inspection basis. The final stage of the test involved the
completion of the Additional Features, Problems and Overall Impression sections of
the Spreadsheet Auditing Software Test Form.
The exception to the aforementioned three-staged approach to testing the software
was SpACE. Unfortunately it was not possible for the author to acquire a copy of
SpACE for testing purposes without attending a training course in its use. As there
was no training course scheduled before the submission date of this report, it was
impossible to test SpACE in the same three staged approach used in the tests of the
other software tools. As a result, it was agreed that the second stage, the test against
the Sample Errors Spreadsheet, would be done by an experienced user of SpACE,
under the observation of the author, with the author awarding results based upon the
observation of the test and completing the Spreadsheet Auditing Software Test Form
33
after the test had taken place. Screenshots and reports were taken from each of the
tests to allow future analysis of the results.
Due to the nature of the errors and the result criteria, along with the fact that the tests
took place over a period of time, the results of the test could be classed as largely
subjective. To attempt to reduce any subjective aspect of these results a normalisation
stage took place once all of the tests had been completed. This involved examining
each error in turn across the five software tools tested and where necessary, adjusting
the results slightly to establish a level of consistency and to reduce the subjective
elements of the results. Although a degree of subjectivity undoubtedly remains in the
test results as they are based upon questions of opinion such as “with this information
in this format, how likely would an auditor be to identify this error? ”, as the expressed
primary aims of the tests were to answer the following questions, it was felt that a
certain level of subjectivity in the test results could be tolerated, particularly as the
results were to be drawn from five different software tools.
•
•
•
•
Are software auditing tools for spreadsheets useful?
How do software auditing tools assist the user in auditing spreadsheets?
What errors are software auditing tools good at identifying?
What errors are software auditing tools poor at identifying?
There follows a summary of the five tests carried out in an attempt to answer the
questions above. The completed Spreadsheet Auditing Software Test Forms can be
found in appendix D of this report, along with a selection of screenshots from each
test.
34
Excel’s Built In Auditing functions
The first software to be tested against the Sample Errors Spreadsheet was Excel’s own
built in auditing functions. These built in functions were unique in that as they are
included in Excel as standard, they were also used to supplement other software tools
tested. As a general rule, the test utilised the built in functionality when testing other
software tools only when the software had indicated potential errors within the cell or
range of cells.
To use the built in functions in Excel, the user needs to unprotect any protected
workbook. This suggests that the tools are primarily aimed at the developer of a
spreadsheet model or a casual user, rather than the end-user of a spreadsheet model.
The built in functions do not allow the auditor to work on a range of cells (with the
exception of functions such as the show formulae function which allows the user to
view a whole spreadsheet as formulae) and so to all intents and purposes the Excel
built in functions provide assistance to a cell by cell inspection rather than attempting
to highlight potential errors.
In the test, which was performed largely on a cell-by-cell inspection basis, the built in
functions successfully highlighted the four ‘formulas overwritten by values’ errors,
using both the view formula mode on the offending worksheets, and by showing that
the cells in question did not contain any precedents. The functions failed to identify
the use of a constant in a formula as a potential error, and had no options to identify
patterns in labels on the report sheet so failed to identify the ‘illogical presentation’
error. As the built in functions required the user to remove worksheet protection
before use, and makes no attempt to identify unprotected cells, the functions failed to
find the ‘incorrect protection’ error. The built in functions had more success on the
three mechanical errors, only struggling on the formula that involved a totalling of
separate rows rather than a whole column. On the first three logical errors (domain
errors) the functions clearly showed the constituents of the formulae, but this gave no
suggestion that the constituents where incorrect. The built in functions fared slightly
better on the final two logical errors (totalling and averaging of percentages) although
it would still be possible for the user to overlook these errors, particularly if the user
did not posses sufficient domain knowledge. The built in functions failed to indicate
any possible errors in the final two errors in the test, both omission errors.
The major drawback of the built in auditing tools in Excel is the fact that in the
majority of cases, the tools are used on an individual cell basis that means that the
auditor is, in effect, doing a cell-by-cell inspection. Whilst this has proven successful
in tests, and the auditing functions provide a more visual representation of the cell
under investigation, a major reason for using a software auditing tool is to reduce the
time spent on auditing spreadsheets by avoiding the time-consuming cell-by-cell
method. The built in tools prove useful in the investigation of single cells and as a
result are used alongside other software auditing tools (SpACE even turns the Excel
auditing bar on if it is not already displayed). The built in functions do not, however,
provide the user with any guidance to identify potentially unsafe cells. It is also
essential to remember that the auditing tools tested are provided free with Excel and
as such do not purport to do as thorough an audit as those provided, usually at an extra
cost, by a third party.
35
Advantages:
Universal
Free
Disadvantages:
Requires cell-by-cell inspections
Passed
Almost Passed
Almost Failed
Failed
Total
0
8
4
5
17
Built In Excel Functions
Passed
Almost Passed
Almost Failed
Failed
36
SpACE
Operis
Analysis Kit
The Excel
Auditor
The
Spreadsheet
Detective
Built-In Excel
Functions
Built in Excel functions results summary
The Spreadsheet Detective
The next software to be tested against the Sample Errors Spreadsheet was The
Spreadsheet Detective. The Spreadsheet Detective is an Excel add-in and is produced
on a shareware or licenceware basis by Southern Cross Software. The full registered
version costs approximately £100. Unlike the test on the built in auditing functions in
Excel, the test of The Spreadsheet Detective was not done on a cell-by-cell inspection
basis.
To function, The Spreadsheet Detective requires all protected workbooks and
worksheets to be unprotected, which, along with the fact that the software does not
provide functions of use to anything other than auditing, suggest s that the software is
designed for use by spreadsheet model developers in the auditing of their own models.
There are two fundamental ways in which the software attempts to assist the user in
the auditing of the spreadsheet. The first is the identification of formula schema which
should mean that the user has to check far fewer formula than under a cell-by-cell
inspection method, as copied formula are identified as belonging to the same schema
and therefore need not be checked. The second is the listing of potential problems
such as references to non-numeric cells or unprotected schema in the footer report.
On the test itself, The Spreadsheet Detective performed particularly well, only really
struggling on the errors that were expected to be difficult to find. The annotate sheet
function which provided an overlay to a worksheet with different types and colours of
shading and text descriptions of formulae meant that the first four errors concerning
formula that were overwritten as values were obvious. The footer report included a
list of potential error cells including the ‘constant in formula’ error so this was also
easy to find. The software failed to highlight the ‘illogical presentation’ error,
although this could come to light when the adjacent cells are examined as they were
deemed to be within a new formula schema as the order had changed. The ‘incorrect
protection’ error was particularly easy to identify as unprotected cells were listed in
the footer report despite the fact that worksheet protection had been disabled. The
Spreadsheet Detective achieved two ‘almost-passed’ and a ‘passed’ result for the
three incorrect summation errors. It achieved the ‘passed’ result because of the
automatic naming function which shows the formula in an English language type
format based upon the row and column headings, which meant that the error resulting
from totalling the wrong rows stood out. The two logical domain errors involving
formulae looking at the wrong columns were also much easier to find due to the
automatic naming function. The total formula that included sub totals and detail lines
error was highlighted in the footer report under references to non numerics as it also
included blank cells, so this was also easy to identify. A combination of the new
schema shaded overlay and the automatic naming function meant that the totalling
and averaging of percentages errors were relatively easy to identify. The lack of any
un-referenced cells identification or label pattern recognition meant that the software
did not highlight the final two omission errors.
In addition to the features used in the test, The Spreadsheet Detective includes
additional features such as formula reports that document all formulae on the
worksheet using the automatic naming function, an inter-worksheet report that shows
37
how data flows between worksheets and a sensitivity report that attempts to highlight
the importance of an individual cells data on the ‘bottom line’ figure.
The Spreadsheet Detective proved successful in the test, and has many functions that
go well beyond those errors in the test spreadsheet. The two pronged approach of
using an overlay to identify formula schema and types of cell, and having a summary
report of potential errors meant that the user was guided towards many of the errors
by the software, the summary report proving particularly useful in identifying the
more subtle errors such as unprotected ranges. The automatic naming of constituents
of formulae worked well in the errors spreadsheet, although this may prove less useful
in a spreadsheet that is not as logically laid out with easy to identify row and column
labels. However, this naming function remains one of the best features of the software
and could even encourage developers to design spreadsheet models in a more logical
way to comply with these naming conventions.
Advantages:
Automatic naming in formulae
Identification of schema
Two-pronged approach using an overlay and a footer report
Production of separate reports for off-line checking
Disadvantages:
Possible confusion due to the many different types of shading
The lack of an option to identify un-referenced cells
38
The Spreadsheet Detective
Passed
Almost Passed
Almost Failed
Failed
39
SpACE
12
2
1
2
17
Operis
Analysis Kit
0
8
4
5
17
The Excel
Auditor
The
Spreadsheet
Detective
Passed
Almost Passed
Almost Failed
Failed
Total
Built-In Excel
Functions
The Spreadsheet Detective results summary
The Excel Auditor
The Excel Auditor was the third software test to be tested against the Sample Errors
Spreadsheet. The software is produced by the fantastically named Big Yellow Gorilla
Software (BYG) and costs approximately £100. The Excel Auditor is an add-in for
Excel and despite its name provides many functions outside the usual scope of
auditing software, which, coupled with the fact that many of its functions are
available even on protected worksheets suggests that the software is designed for use
throughout the life cycle of a spreadsheet.
The Excel Auditor provides two primary and two secondary auditing tools. The
primary tools being the Audit Map, which provides a traditional audit map of a
worksheet, and the Worksheet Cell Analyser, which documents the contents of a
worksheets cells. Both of these tools produce a report on a separate workbook. The
secondary tools are the Cell Roots and Circular Reference Analyst functions.
On the test itself, The Excel Auditor performed poorly. Unlike the test of Excel’s built
in auditing functions, The Excel Auditor was not tested on a cell-by-cell inspection
basis. This was because a primary aim of a supplementary auditing package should be
to save the auditor time that would be taken if a cell-by-cell inspection were
necessary, and also to keep the test consistent with the other supplementary tools
tested in this report. Unfortunately this meant that The Excel Auditor did not perform
as well as the Excel functions due to the method of testing.
As a cell-by-cell inspection was not used with The Excel Auditor, the first four errors,
concerning formulae overwritten with values, whilst clearly shown on the audit map,
could easily be overlooked as they would require the use of the audit map alongside
the original worksheet. In this case it could be argued that it is easier to move the
cursor over each cell in the worksheet and watch the Excel input bar. As a result, The
Excel Auditor only achieved an ‘almost-failed’ result for each of these errors. The
Excel Auditor also achieved an ‘almost-failed’ result for the ‘constant in formula’
error as whilst it clearly showed the constituents of the formula in the worksheet cell
analyser, it failed to warn of any potential problems with this approach. The software
failed to identify any problem with the ‘illogical presentation’ error and whilst the
documentation function showed which worksheets had protection enabled, it could
not do the same for individual cells. On the three mechanical errors that related to
incorrect summation, The Excel Auditor, whilst clearly documenting the formulae
concerned, failed to suggest any problems with the formulae and therefore only
achieved an ‘almost-failed’ result for each of these errors. The same issues surfaced
on the two logical/domain errors concerning ‘incorrect formula due to the wrong
column being used’ and the summation formula that included sub total rows as well as
detail rows. Both of the incorrect percentage calculations suffered from the fact that
they were buried away in the 64 page worksheet analysis report produced for the
report sheet, although the average calculation was slightly easier to identify as an
error. The final two omission errors both failed to be identified by The Excel Auditor
as there is no way of looking for label patterns of efficiently looking for cells with no
dependents.
40
Perhaps the biggest problem with The Excel Auditor as an auditing tool is that is still
requires cell-by-cell inspection to allow the auditor to confidently audit the
spreadsheet. The software is also let down by the lack of visual aids to auditing,
particularly by the fact that it functions on separate reports meaning that the user is
required to jump from report to spreadsheet whilst auditing the spreadsheet. The
Excel Auditor does however provide useful features to document the spreadsheet
which, although not strictly an auditing feature, should help to encourage good
spreadsheet design and theoretically therefore more accurate spreadsheets. The Excel
Auditor is more likely to be of use as a documentation tool rather than an auditing tool
as the documentation tools, although not covered in the test , are quite useful and easy
to use.
Advantages:
Clear reports to assist in off -line cell-by-cell inspection approach
Good documentation tools
Powerful search option
Disadvantages
No identification of schema
No attempt to identify potentially problematic cells
Reports produced separately rather than overlaid on a copy of the worksheet
Requires a cell-by-cell inspection approach to be used correctly
41
12
2
1
2
17
0
1
12
4
17
The Excel Auditor
Passed
Almost Passed
Almost Failed
Failed
42
SpACE
The Excel
Auditor
0
8
4
5
17
Operis
Analysis Kit
The
Spreadsheet
Detective
Passed
Almost Passed
Almost Failed
Failed
Total
Built-In Excel
Functions
The Excel Auditor results summary
The Operis Analysis Kit
The fourth software tool to be tested against the Sample Errors Spreadsheet was the
Operis Analysis Kit. This software is available from Operis Business Engineering
Limited at a cost of approximately £465 and is in the form of an Excel add-in. As with
most of the other tools tested, Operis Analysis Kit requires worksheet protection to be
disabled before most of the functions can be used, which along with functions such as
the add version documentation function and the extra naming functions suggests that
the software is aimed primarily at the developer of spreadshe et models. As with The
Excel Auditor, Operis Analysis Kit includes many functions that emulate and enhance
functions that already exist in Excel.
To perform spreadsheet auditing, Operis Analysis Kit adopts a two-pronged approach.
The first allows the user to search the worksheet for particular types of cells that are
more likely to be problematic, such as those with no dependents or those with formula
containing hardwired constants. The second approach concentrates on graphically
mapping the cells on a worksheet overlay to identify formula schema, referred to in
Operis Analysis Kit as distinct formula, and types of cell, and to document the distinct
formula and relevant statistics.
In the test itself, the four ‘formulas overwritten with values’ errors were easily
identified using the option to overlay colour-coded cells to the worksheets. The search
for hardwired constants option correctly identified the ‘fixed value in a formula’ error.
The software did not, however, highlight any potential problem with the ‘illogical
presentation’ error, although this could be spotted when the adjacent formula cells
were examined as they were deemed to have distinct formula, so an ‘almost-failed’
result was recorded for this test. Operis Analysis Kit failed the ‘incorrect protection’
error, as it makes no attempt to identify which cells are left un-protected. The three
mechanical errors all achieved an ‘almost -passed’ result as each was identified as a
distinct formula and the contents of these formula were clearly documented. The two
logical/domain errors achieved ‘almost-failed’ results, as although the formula was
marked as distinct in each case, and the formulae were clearly documented, there was
no suggestion that the formulae were incorrect. The ‘incorrect summation of a column
including sub totals’ error was easily identified using the search option to find
references to blank cells. The two incorrect percentages tests produced contrasting
results. On the first of these tests, which involved the totalling of percentages, Operis
Analysis Kit achieved a ‘failed’ result as the formula was not identified as distinct as
it matched the formula in the cell to the left, whereas on the second test, that of the
averaging of percentages, Operis Analysis Kit achieved a ‘passed’ re sult as this was
correctly identified as a distinct formula and upon examination of the cell
documentation was obvious. The two omission errors also produced contrasting
results; with the first error achieving a ‘failed’ result, as there were no options to
identify label patterns, whereas the second error achieved a ‘passed’ result as using
the search for un-referenced cells on the file worksheet correctly highlighted the
problem.
Although Operis Analysis Kit performed quite well in the test, one particular problem
came to light. This was that when identifying distinct formulae, the software relies
upon the formula logically matching the formula in an adjacent cell. As a result
43
Operis Analysis Kit identified ‘new’ distinct formula unnecessarily in cases where
‘blocks’ of formula were separated by blank lines on the report sheet or detail lines on
the import sheet. This could of course lead to duplication of work. Operis Analysis
Kit as a whole is very easy to use thanks to its operation via a simple additional menu
in Excel. It is deceptively powerful, particularly in the search options, and managed to
at least hint at all of the more ‘findable’ errors with the exception of its inability to
identify unprotected cells.
Advantages:
Easy to use
Powerful search options
Two pronged approach of overlay and report
Disadvantages:
Limitation in schema recognition
The Excel
Auditor
Operis
Analysis Kit
0
8
4
5
17
12
2
1
2
17
0
1
12
4
17
8
3
3
3
17
Operis Analysis Kit
Passed
Almost Passed
Almost Failed
Failed
44
SpACE
The
Spreadsheet
Detective
Passed
Almost Passed
Almost Failed
Failed
Total
Built-In Excel
Functions
Operis Analysis Kit results summary
Spreadsheet Audit for Customs and Excise (SpACE)
The final software tool to be tested against the Sample Errors Spreadsheet was
SpACE. This software was developed in -house by HM Customs and Excise for use by
VAT inspectors in auditing clients spreadsheets, but is now available to the public at a
cost of £100 + VAT. Due to its history SpACE is almost exclusively an auditing tool
with a slight emphasis towards the calculation of tax liabilities. SpACE uses a ‘bruteforce’ approach to remove any passwords in a workbook before copying all the
worksheets in a workbook to a new workbook, meaning that the auditor is working on
a copy of the original workbook. This confirms the view that SpACE is for use by
spreadsheet auditors rather than developers. SpACE works by using a combination of
search facilities, overlaid mapping options and the identification of unique formula to
attempt to highlight potential errors in a spreadsheet.
The test of SpACE was done slightly differently to the other software tool tests as the
test was performed by Ray Butler, an employee of HM Customs and Excise and a
regular user of the SpACE software, under the observation of the author. SpACE
proved exceptionally good on the test itself, recording at least an ‘almost-passed’
result on each error test. The software easily passed the first four ‘formula replaced
with values’ tests using either the colour coded overlay or the identification of
numeric cells with no dependents option. SpACE achieved an ‘almost-passed’ result
in the ‘fixed value in formula’ error as the software allows the user to search for fixed
values, but the user has to enter the value to be found. SpACE achieved an ‘almostpassed’ result on the ‘illogical presentation’ error as it allowed the user to search for
particular text strings and apply colour coding which, along with the indication that
the adjacent formula cells are out of sync with the ot her should highlight the problem.
SpACE successfully found the ‘incorrect protection’ error as despite automatically
unprotecting the worksheets, the software is still able to identify the individual cells
that have been marked as unprotected. In line with most of the other tools tested,
SpACE correctly identified the three mechanical errors (incorrect summation errors)
as unique formula yet the lack of a visual report means that this error could be
overlooked by the user. The two logical/domain errors both achieved an ‘almostpassed’ result, thanks largely to the SpACE option of attempting to link user specified
root and bottom-line cells. When the software failed to establish a link between these
two cells, further investigation revealed the errors. The ‘incorrect summation of
column containing subtotals’ error was easily found by SpACE as it was highlighted
as a unique formula, which upon further investigation shows the error to be obvious.
Both of the erroneous percentage calculations were highlighted by SpACE in a way
that was obvious enough to earn a ‘passed’ result. SpACE was the only software tool
that had any success with the first omission error, which involved a row missing from
every sheet in the workbook. It achieved an ‘almost-passed’ result by a combination
of a schema identification method that allows ‘blocks’ of formulae to be grouped
together and the option to highlight text strings with different coloured backgrounds.
The final omission error achieved a ‘passed’ result on SpACE as the option to identify
cells with no dependencies highlighted the offending row on the file worksheet.
SpACE is a ‘tried and tested’ software tool and has obviously been in use and subject
to improvement for some time. In addition to the functions covered in the test, SpACE
includes more in-depth auditing tools such as the ability to check lists of data for
45
duplicates and attempt to identify the numbers that make up a particular total. The
only major function missing from SpACE that was present in any of the other
software tools tested appears to be some form of English language type formula
description tool similar to that found in The Spreadsheet Detective. The power and
complexity of the software is reflected in the fact that its use is included in a two day
course that users can attend before they begin to use it at a cost of £450 + VAT.
Advantages:
Good schema identification
Root and bottom line cell link identification
Works in a copy of the original spreadsheet
Disadvantages:
Lack of an English language naming system
The
Spreadsheet
Detective
The Excel
Auditor
Operis
Analysis Kit
SpACE
Passed
Almost Passed
Almost Failed
Failed
Total
Built-In Excel
Functions
SpACE results summary
0
8
4
5
17
12
2
1
2
17
0
1
12
4
17
8
3
3
3
17
9
8
0
0
17
SpACE
Passed
Almost Passed
Almost Failed
Failed
46
Results
Results by Software Tools
Total Score by Software
50
40
Built-In Excel Functions *
30
The Spreadsheet Detective
20
Operis Analysis Kit
The Excel Auditor
SpACE
10
0
As can be seen in the table and chart above, SpACE and The Spreadsheet Detective
lead the way in the test results, scoring 43 (84%) and 41 (80%) respectively. It is
possibly significant however that the SpACE results were assisted by the fact that the
47
test was carried out by an experienced SpACE user (although the results were
allocated by the author), rather than by the author, who was new to all of the other
software before the tests began. It should also be noted that even on a cell-by-cell
inspection basis, the built in Excel functions only score 24 (47%), which suggests that
all is not well and that the need to go beyond the built in auditing functions exists.
Operis Analysis Kit, which is probably the easiest tool to use, scored a respectable 33
(65%), only letting itself down in a few areas such its inability to identify unprotected
cells and limited schema identification method. The Excel Auditor only scored 14
(27%), although this was largely down to the method of testing as it needs to be used
as a tool to assist in a cell-by-cell inspection, whereas it was tested on the same as the
other three add-in packages.
48
6
6
6
6
6
6
4
6
4
4
4
6
4
4
6
6
6
6
4
6
SpACE
4
4
6
4
4
4
4
6
6
6
Operis
Analysis Kit
The
Spreadsheet
Detective
6
6
6
4
6
6
6
6
6
6
Feature
Schema/Unique cell identification
Potential Errors Highlighted in Report
Option to search for Potential Errors
Visual/Graphical overlay on worksheet(s)
Formula assigned English language names
Highlights unprotected data cells
Summary/Statistical Report
Root cell to bottom line cell link
Software includes documentation tools
Software includes development tools
The Excel
Auditor
Built-In
Excel
Functions
During the test, it became apparent that certain functions were more useful than
others, both in the Sample Error Spreadsheet test itself, and in the familiarisation
stage of the test where other functions were investigated. Possibly the most essential
ability for a software tool to possess is the ability to recognise formula schema. It is
this ability that lifts the software from being a tool to assist in a cell-by-cell
inspection, to a tool that means a time-consuming cell-by-cell inspection can be
avoided. A second function that could be classed as essential is the ability to provide
the user with a visual overlay on the worksheet, identifying different schema/different
types of cells etc. Falling into the ‘very useful’ category are functions such as
supplementary potential error cells reports, the option to search for particular cells
that are prone to errors such as constants in formulae or un-referenced cells, the
ability to identify unprotected cells, and the use of English language formula
descriptions based upon column and row labels. It is in these functions that SpACE
and The Spreadsheet Detective excel, and that enabled them to score so highly in the
tests. Other features such as documentation and development tools, whilst not
contributing to the test results due to the nature of the Sample Errors Spreadsheet,
could certainly be useful, depending upon who the user of the software is and the role
they perform i.e. developer/auditor/end-user. The main features of the software tools
tested are summarised in the table below:
4
4
4
4
6
4
4
4
6
6
Average Score for
type of error excl.
Excel built In
Qualitative
Mechanical
Logical
Ommission
Average Score for
type of error
Results by Type of Error
2.0
1.8
2.0
0.8
2.1
1.8
2.1
1.0
When the results of each of the software tests are combined, it is possible to gain an
overview of the levels of success that was achieved in relation to each type of error.
From the results of the test, summarised in the table above, it is obvious that the
software tools tested performed to a reasonable standard on all but the omission
errors, for which the average result was 0.8, far below the average results reported for
the other three error types. This is not altogether surprising as these errors are the type
that an auditor is more likely to identify with the human eye than with a software tool.
Those tools that highlighted these omission errors relied upon non-referenced cells
elsewhere in the spreadsheet model, and identifying label/formula patterns. The other
three error types all achieved good results in the tests, although the mechanical errors
did not quite match the qualitative and logical errors. Perhaps the most surprising
result was the success of the qualitative errors, although these were improved slightly
by having a particularly easy to spot (yet quite common) error in four different places.
Considering that a definition of a qualitative error is that it does not (yet) have an
impact on the bottom line figures, it was anticipated that these would be particularly
difficult to find. The results on all but the omission errors do suggest however that
software tools can have a positive effect on software auditing detection and
efficiency.
49
Results by Individual Errors
Average Result
Average Results for Individual Errors
3.0
2.5
2.0
1.5
1.0
0.5
0.0
1 2 3 4 5 6 7 8 9 1011121314151617
Error Number
Formulas overwritten with values (errors 1 to 4)
On these errors, the results averaged 2.6, a particularly successful result that
illustrated how easy these errors can be to detect. Only The Excel Auditor of the addin software tested failed to achieve a ‘passed’ result. The success in finding these
errors was largely down to the colour -coded overlays that could be applied to the
worksheet making these errors stand out.
Fixed value used when a named cell reference should be used (error 5)
This error achieved a respectable result of 1.8, which when the cell-by-cell inspection
method used in the built in Excel functions test was removed, increased to 2.3. Both
The Spreadsheet Detective and Operis Analysis Kit explicitly highlighted this error
via an error report and a search option respectively. SpACE was close behind with a
search option but this option needed the value to be specified by the user.
Presentation Illogical (error 6)
The presentation illogical error proved very difficult to detect, with SpACE getting
the closest by allowing certain text strings to be formatted with a different background
colour meaning that text patterns were easier for the user to identify. Other than this,
Operis Analysis Kit and The Spreadsheet Detective both partially identified this error
by virtue of the fact that the adjacent cells had new formula schema.
50
Incorrect Presentation (error 7)
This error surprisingly proved quite difficult to identify in the tests. However both
The Spreadsheet Detective and SpACE correctly identified the unprotected range of
cells despite having previously disabled the worksheet protection. The Excel Auditor,
Operis Analysis Kit and the built in Excel functions all failed to identify the erroneous
cells, although The Excel Auditor did indicate which worksheets were protected.
Incorrect Summation (errors 8 to 10)
These errors received a reasonable average result of 1.8. The more successful methods
of finding these errors were the graphical overlays, which showed new formula
schema and/or highlighted the cells being totalled. The exceptional result on these
errors was The Spreadsheet Detective highlighting the incorrect row error (error 9) by
using the labels to indicate, in English that the values being totalled were not from the
correct rows.
A formula copied down a colu mn looks at the wrong column (errors 11 and 12)
These errors were designed to be difficult to detect and the result of 1.6 for each was
higher than expected. Although each of the software tools were capable of
highlighting the formula and displaying the formula in a readable style, the nature of
this error is such that even when the formula is displayed clearly, the error is not
always obvious without some domain knowledge. The only software tool to achieve a
‘passed’ result in this test was The Spreadsheet Detective. This was largely down to
the English descriptions. SpACE achieved on ‘almost-passed’ thanks to its failure to
establish a link between the source data and the bottom line cell, which prompted
further investigation that subsequently detected the error.
Incorrect Summation of a column including subtotals (error 13)
Whilst a visual overlay of this formula showed which cells were included in the
formula, the nature of the error meant that this error could still be overlooked.
‘Passed’ results were achieved by The Spreadsheet Detective, Operis Analysis Kit and
SpACE by highlighting the cell as a prospective error due to the fact that it contained
blank cells.
Percentages added rather than calculated on a total cell (error 14)
With a graphical representation of the formula in question this error is relatively easy
to detect. The Spreadsheet Detective and SpACE both achieved a ‘passed’ result on
this error by virtue of identifying a unique formula which when viewed with the built
in Excel functions became obvious. Operis Analysis Kit however failed this test as the
total cell to the left was the same logical formula and so the cell was not highlighted
as a distinct formula.
51
Percentages averaged rather than calculated on a total cell (error 15)
Although superficially similar to the previous error, this error proved to be much
easier to detect. The fact that it was an average of percentages made the error easier to
detect even on a formula report. This error was also highlighted as a distinct formula
in Operis Analysis Kit meaning that this achieved a ‘passed’ result on this test,
whereas a ‘failed’ result was achieved by Operis Analysis Kit on the previous error.
A row missing from all of the worksheets (error 16)
The omission errors were deliberately included as a difficult test for the software
tools. The first was designed to provide an error that was expected to be much more
likely to be detected by the human eye than a software tool. The fact that the row in
question was removed from all of the worksheets means that the software would have
to have some way of identifying label patterns to identify the absence of the row.
Only SpACE came close to identifying the absence of the row, thanks to a
combination of text background colour coding and the ability to recognise unique
formula across ‘blocks’ of formulae.
A row missing from one of the worksheets although it exists on the others (error 17)
This omission error was slightly easier to find than the previous error as the row of
data deleted from the Report worksheet still existed on the File worksheet. Both
SpACE and Operis Analysis Kit achieved a ‘passed’ result on this test due to the
option to highlight cells with no dependents.
52
Test Conclusions
At the outset of the test, the following four questions were asked:
•
•
•
•
Are software-auditing tools for spreadsheets useful?
How do software-auditing tools assist the user in auditing spreadsheets?
What errors are software-auditing tools good at identifying?
What errors are software-auditing tools poor at identifying?
The test has successfully provided the following answers to these questions:
•
Are software-auditing tools for spreadsheets useful?
The tests proved conclusively that spreadsheet auditing software tools can be useful.
The more successful software achieved detection rates of over 80%. However it must
be remembered that software tools do not detect and correct errors, they merely point
the user in the right direction.
•
How do software-auditing tools assist the user in auditing s preadsheets?
The software tools used various different methods of identifying potential errors,
however the most successful used a combination of the following four features:
Schema Identification
Visual Methods
Search for/Report on potential error cells
Spreadsheet documentation
The software tools tested tended to be designed as add-ins to Excel, and as such could
be used alongside the built in Excel Auditing functions.
•
What errors are software-auditing tools good at identifying?
Software tools proved to be particularly good at detecting qualitative, quantitative
logical and quantitative mechanical errors in the test.
•
What errors are software-auditing tools poor at identifying?
Software tools proved somewhat less successful at detecting quantitative omission
errors. Given the nature of these errors, this was not a surprising result.
53
Overall Conclusions
Conclusions From Project
It is obvious from the research in the first part of this report that spreadsheet errors are
still a major problem. Research has repeatedly found that rates of spreadsheet errors
are in line with those in more traditional programming techniques, yet the general
perception of spreadsheets is that they are much simpler than traditional
programming. It is because of this that spreadsheets are frequently not subjected to the
same strict testing disciplines accepted as standard procedure in traditional
programming. In the authors professional experience as a developer of a number of
spreadsheet models, spreadsheets are not allocated as much time and resources as a
traditional programming language would be for testing and auditing purposes. Cellby-cell inspection is largely unheard of and even dummy data testing can be limited.
These experiences are confirmed as commonplace by the academic research
published, thus confirming that the author’s experiences are not unique. Research has
taken place, which has found that spreadsheet auditing is easier using a visual
approach. However to date there has been no published research into the usefulness
and methods used in spreadsheets auditing software tools. The second part of this
report provides an initial investigation into software tools. The evidence shows that
software tools are definitely useful in the detection of software errors, although they
perform poorly on omission errors. The software tools potentially have three methods
of assisting in the audit of a spreadsheet. The first is by applying a visual approach to
auditing to the user and the second is to identify cells that potentially contain errors.
The third method available to software tools, and potentially the most powerful, is the
identification of formula schema, which allows the user to adopt a ‘focused cell
inspection’ approach to spreadsheet auditing rather than a traditional cell-by-cell
inspection approach.
Personal reflections
During the course of this project many skills were acquired and/or developed by the
author. At the most obvious level, the author, whilst not new to spreadsheets as he has
worked with Lotus 1-2-3 for the last 5 years, was new to Excel and needed to become
proficient in Excel before designing the Sample Errors Spreadsheet, and subsequently
using this in the tests. The author was completely new to all of the software auditing
tools that were tested and therefore had to achieve a good understanding of the way in
which these tests worked and how they were used. As the author was completing the
final year of a three -year degree program, skills involved in researching, reading and
summarising academic reports had been acquired, however there is no doubt that a
project of this magnitude has meant that these skills have been developed further.
The author experienced very few major problems during the course of this project.
Perhaps the largest problem encountered on the first half of the project was the
incredible amount of research that had to be read through which did not offer anything
new to the subject, rather it was the type that merely concludes that the author(s)
54
generally concur with papers that had been written previously. The only problem on
the second half of the project was the inability to attend a SpACE training course,
which led to the SpACE software being tested by a trained SpACE user, under
observation of the author. However, given the rather subjective nature of the test and
the fact that the results were subsequently subjected to a normalisation process, this
should not have had a bearing on the overall conclusions of the tests.
At the outset of this project, the project proposal defined the following as the five
primary objectives for this project:
•
•
•
•
•
Investigate, summarise and analyse the research to date on spreadsheet errors.
Investigate, summarise and analyse research into how developers try to avoid
software errors from occurring.
Investigate, summarise and analyse how spreadsheets are audited.
Identify and review the software available to assist in spreadsheet auditing.
Apply the spreadsheet auditing methods and software tools identified to either a
standard custom built spreadsheet, or a series of spreadsheet models to investigate
their impact on error detection/correction.
The first three of these objectives were achieved in the first part of the report, which
documented much of the research to date on these three subjects and drew various
conclusions. The fourth and fifth primary objectives, of identifying and reviewing
software tools available and applying these to a standard custom built spreadsheet
were achieved in the second part of the report. The second part of the report also,
during the course of the test, enabled conclusions to be drawn concerning the ways in
which software tools work in auditing spreadsheets and the usefulness of each of the
methods used.
The project proposal also achieved a secondary objective:
•
To test the different methods/software tools available against a standard
spreadsheet in an experimental environment with a number of different subjects.
This objective was dropped at the interim report stage as it had become apparent that
no research had been done into the general ways spreadsheet auditing software works
and it was essential therefore to investigate and document this before testing the
software in a more open environment. The test outlined in this objective is however
still relevant, particularly now that this report has been completed, and as such would
now prove useful. Unfortunately time constraints do not allow the pursuit of this
objective within this report, although drawing upon the research documented in this
report means that this could now be done. This objective is therefore included in the
Areas for further study section of this report.
In the project, a Gantt chart was constructed to assist in the planning and scheduling
of this project. At the time of the interim report, this pla n was revised. At the
submission of the interim report, the only tasks that should have been completed were
the investigation of error research sections and the spreadsheet lecture attendance.
Each of these tasks was completed on time. The revised schedule , published in the
interim report, had changed to reflect the change in focus of the project as a whole, in
55
that it no longer included a test using different subjects, instead the onus for the tests
fell directly on the author.
This revised schedule for the second part of the project is shown below:
ID
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
Task Name
Submit Interim Report
Write up spreadsheet history
Write up spreadsheet error research
Write up spreadsheet error impact
Write up spreadsheet auditing
research/methods
Develop/Acquire sample spreadsheet(s)
for testing
Test Spreadsheet Auditing Assistants
against sample spreadsheet(s)
Write up test results
Review/Conclude tests
Write up conclusions
Conclude the project
Academic review
Write up final report
Prepare presentation
Do presentation
Submit Final Report
Write paper for EuSpRIG conference
Present paper at EuSpRIG conference
Duration
0 days
28 days
28 days
28 days
Start
Thu 01/02/01
Mon 15/01/01
Mon 15/01/01
Mon 15/01/01
Finish
Thu 01/02/01
Wed 21/02/01
Wed 21/02/01
Wed 21/02/01
28 days Mon 15/01/01 Wed 21/02/01
7 days Thu 01/02/01
Fri 09/02/01
10 days Mon 12/02/01
Fri 23/02/01
7 days
4 days
7 days
7 days
4 days
14 days
14 days
0 days
0 days
21 days
0 days
Mon 26/02/01
Wed 07/03/01
Tue 13/03/01
Thu 22/03/01
Mon 02/04/01
Fri 06/04/01
Tue 13/03/01
Mon 02/04/01
Tue 01/05/01
Tue 01/05/01
Thu 05/07/01
Tue 06/03/01
Mon 12/03/01
Wed 21/03/01
Fri 30/03/01
Thu 05/04/01
Wed 25/04/01
Fri 30/03/01
Mon 02/04/01
Tue 01/05/01
Tue 29/05/01
Thu 05/07/01
In a review of the revised plan, it is apparent that the estimates for some tasks varied
greatly from the actual time taken to perform the table: for example, the
Develop/Acquire sample spreadsheets for testing, took approximately twice the 7
days estimated. The irregular times available for work on the project (due to other
work commitments and lack of holidays before the new CWS holiday year in April)
also meant that the dates became guidelines rather then strict deadlines. The plan did,
howeve r, serve its purpose as a checklist of tasks that needed to be completed to
complete the project.
With any project, the experience gained from doing the project means that in
retrospect, the project would be done differently. This project was no different. Whilst
the author is generally pleased with the project, the answer to the perennial question
“If you could do it all again, what would you do differently”? does provide a few
56
changes. The author would, in this hypothetical situation, definitely place less
emphasis on the research section of the project and more emphasis on the test part. In
addition to this being the far more interesting section to do, there is also the fact that
once the first few papers in each field are investigated, the general points have been
made, with subsequent reports merely adding evidence to these points. Other changes
the author would make would largely be based upon making minor changes to the
Sample Errors Spreadsheet such as changing it to a more generic type so that it
covered more than just financial spreadsheet types of errors or removing some of the
repeated errors which could have too much impact on the results. The author would
also attempt to find a way of making the test more objective, although this was
achieved to a certain degree with the normalisation process.
57
Areas For Further Study
The first part of this report provided evidence that extensive research into spreadsheet
errors has taken place. This research has covered identifying and categorising errors,
quantifying errors, methods of designing spreadsheets to avoid errors and error
detection. There is not however, very much research into the use of software tools to
detect errors in spreadsheets.
The second part of this report attempts to redress this by testing a series of software
tools against a spreadsheet with seeded errors. Whilst the tests proved that spreadsheet
auditing software is useful. Certain limitations in the software suggest that further
study in this field would be beneficial. The nature of the test meant that the results
were largely subjective. The test was primarily carried out by the author, who also
designed the sample errors spreadsheet. This meant that the author had to speculate
how much user interaction would be needed to identify an error and award a result
accordingly. A more objective test would be to have the test carried out by a group of
people who did not know the location of the errors before the tests. Another limitation
of the test was that the sample errors spreadsheet was based primarily on a financial
report model. Spreadsheets are used for many different applications other than
financial applications. It would be useful therefore to perform the tests on a number of
spreadsheets from different fields such as a results analysis spreadsheet or a ‘what if’
based spreadsheet.
58
Appendix A – References
World Wide Web Pages
Binning, B. (1999) ‘Spreadsheets’
http://backofficesystems.com/tips/apps/spreadsheet.htm
Accessed 5/10/00
Bricklin, D.
‘Was VisiCalc the “first” spreadsheet?’
www.bricklin.com/history/
Accessed 27/11/00
Power, D. J.
‘A Brief History of Spreadsheets’
http://www.dssresources.com/history/sshistory.html
Accessed 16/10/00
Netfuture on-line newsletter (Issue 54, 30/7/97) ‘Spreadsheet Errors’
http://www.oreilly.com/people/staff/stevet/netfuture/1997/Jul3097_54.html
Accessed 24/10/00
Woodburg, G.G. (1989) ‘Re: ’Computer Error’ in Durham N.C. Election Results’
The Risks Digest (9:42)
http://catless.ncl.ac.uk/Risks
Accessed 3/10/00
Published/Academic Documents
Business Week (24/9/84) ‘How Personal Computers Can Trip Up Executives’ p94102
Butler, R (1997) ‘The Subversive Spreadsheet’
Protocol (ISACA Wellington NZ) reprinted in DataWatch (ISACA London UK)
Butler, R. (2000), ‘Risk Assessment for Spreadsheet Developments: Choosing which
Models to Audit’
EuSpRIG
Cleary, P.M. (2000) ‘How Important are Spreadsheets to Organisations?’
European Spreadsheet Risks Interest Group
‘Spreadsheet Risks, Audit and Development Methods’
Symposium Proceedings
EuSpRIG
59
Creely, P., ‘Review of Errors in Spreadsheets’
Final Year BSc Paper, University of Salford, 2000
Chadwick, D., Rajalingham, K., Knight, B., Edwards, D. (1999), ‘An Approa ch to the
Teaching of Spreadsheets using Software Engineering Concepts’
Information Integrity Research Centre
School of Computing and Mathematical Sciences
University of Greenwich
Chadwick, D., Knight, J. Clipsham, P. (1997), ‘Information Integrity In End User
Systems’
Proceedings of the IFIP TC-11 WG 11.5 1st working Conference on Integrity and
Internal Control in Information Systems, Zurich, Switzerland
Chadwick, D., Knight, J. Rajalingham, K. (2000) ‘Quality Control in Spreadsheets: A
Visual Approach Using Colour Coding To Reduce Errors In Formulae’
Information Integrity Research Centre
School of Computing and Mathematical Sciences
University of Greenwich
Chen, Y. & Chuan Chan, H. (2000a) ‘Visual Checking of Spreadsheets‘
School of Computing
National U niversity of Singapore
European Spreadsheet Risks Interest Group
‘Spreadsheet Risks, Audit and Development Methods’
Symposium Proceedings
EuSpRIG
Chen, Y. & Chuan Chan, H. (2000b) ‘An Exploratory Study of Spreadsheet
Debugging Processes’
The Pacific Asia Conference on Information Systems
June 2000
Computing Magazine (2/3/95), ‘Suppliers admit to software errors’, p11
Ditlea, S (1987) ‘Spreadsheets can be hazardous to your health’
Personal Computing Magazine (11:1) January 1987 p60-69
ICAI Factsheet Series on IT
Factsheet 1 – ‘Spreadsheet Modelling’
Knight, B., Chadwick, D., Rajalingham, K., (2000) ‘A Structured Methodology for
Spreadsheet Modelling’
University of Greenwich
Information Integrity Research Centre
Panko, R. & Halverson, R, (1996) ‘Are Two Heads Better Than One? (At Reducing
Errors in Spreadsheet Modelling)
Office Systems Research Journal
60
Panko, R. & Halverson, R, (1996) ‘A Survey of Research on Spreadsheet Risks’
Proceedings of the 29th Hawaii International Conference on System Sciences
Panko, R. (1996) ‘Hitting the Wall: Errors in Developing and De-bugging a ‘Simple’
Spreadsheet Model’
Proceedings of the 29th Hawaii International Conference on System Sciences, Maui,
Hawaii. January 2-5 1996
Panko, R. (1998) ‘What we know about Spreadshe et Errors’
Journal of End User Computing’s Special Issue on Scaling Up End User Development
Volume 10, Number 2, Spring 1998 p15-21
Panko, R. (1999), ‘Applying Code Inspection to Spreadsheet Testing’
Journal of Management Information Systems, 16(2) Fall 1999, p159-176
Panko, R. (2000a), ‘Spreadsheet Errors: What We Know. What We Think We Can
Do’
Proceedings of the Spreadsheet Risk Symposium
EuSpRIG
Greenwich
July 17-18, 2000
Panko, R. (2000b) ‘Two Corpuses of Spreadsheet Errors’
Proceedings of the 33rd Hawaii International Conference on System Sciences
Ragsdale, C. & Conway, D. (1997), ‘Modelling optimisation Problems In The
Unstructured World Of Spreadsheets’
The International Journal of Management Science
Volume 25, Number 3, p313-322, 1997
Rajalingha m, K. & Chadwick, D. (1998), ‘Integrity Control of Spreadsheets:
Organisation and Tools’
Proceedings of the 2nd Annual IFIP TC-11 WG 11.5 Working Conference on Integrity
and Internal Control in Information Systems Bridging Business Requirements and
Researc h Results, Virginia, USA
Rajalingham, K., Chadwick, D., Knight, B., Edwards, D. (1999), ‘Efficient Methods
for Checking Integrity: An Integrated Spreadsheet Engineering Methodology (ISEM)’
Information Integrity Research Centre
School of Computing and Mathematical Sciences
University of Greenwich
Rajalingham, K, Chadwick D, Knight, B (2000a) ‘Classification of Spreadsheet
Errors’
European Spreadsheet Risks Interest Group
‘Spreadsheet Risks, Audit and Development Methods’
Symposium Proceedings
EuSpRIG 2000
61
Rajalingham, K., Chadwick, D., Knight, B., Edwards, D. (2000b), ‘Quality Control in
Spreadsheets: A Software Engineering-Based Approach to Spreadsheet Development’
Proceedings of the 33rd Hawaii International Conference on System Sciences, 2000
Reed, N. & Batson, J. (1999), ‘Spreadsheet Modelling Best Practice’
Business Dynamics
PriceWaterhouseCoopers
The Daily Telegraph (23/7/98) ‘Computer error costs graduate job’ p1
62
Appendix B – The Sample Errors Spreadsheet
The following pages contain the follow ing views from the Sample Errors
Spreadsheet:
•
•
•
•
•
•
Input Sheet – Normal View
Input Sheet – Formula View
Data Sheet – Normal View
Data Sheet – Formula View
Report Sheet – Normal View
Report Sheet – Formula View
63
Year
Enter this weeks flash turnover figures in the higlighted cells.
Region
SCOTLAND
SCOTLAND
SCOTLAND
SCOTLAND
SCOTLAND
SCOTLAND
NORTHERN IRELAND
NORTHERN IRELAND
NORTHERN IRELAND
NORTHERN IRELAND
NORTHERN IRELAND
NORTHERN IRELAND
GREATER NOTTINGHAM
GREATER NOTTINGHAM
GREATER NOTTINGHAM
GREATER NOTTINGHAM
GREATER NOTTINGHAM
GREATER NOTTINGHAM
SOUTH MIDLANDS
SOUTH MIDLANDS
SOUTH MIDLANDS
SOUTH MIDLANDS
SOUTH MIDLANDS
SOUTHERN
SOUTHERN
SOUTHERN
SOUTHERN
SOUTHERN
SOUTHERN
Week
2000
Category
This Week
FOOD ONGOING
3010154
FOOD NEW
191454
NON FOOD ONGOING
482457
NON FOOD NEW
33410
CLOSED
72514
TOTAL
3789989
FOOD ONGOING
439541
FOOD NEW
28874
NON FOOD ONGOING
71454
NON FOOD NEW
0
CLOSED
9894
TOTAL
549763
FOOD ONGOING
2414574
FOOD NEW
178485
NON FOOD ONGOING
441521
NON FOOD NEW
55810
CLOSED
61004
TOTAL
3151394
FOOD ONGOING
989414
FOOD NEW
0
NON FOOD ONGOING
104105
CLOSED
62451
TOTAL
1155970
FOOD ONGOING
3214574
FOOD NEW
225484
NON FOOD ONGOING
610484
NON FOOD NEW
58741
CLOSED
39584
TOTAL
4109283
Input Sheet – Normal View
64
Inflation
20
2.50%
Input Sheet – Formula View
65
Report Line No Year Week Region
FT31
2 2000
19 SCOTLAND
FT31
3 2000
19 SCOTLAND
FT31
4 2000
19 SCOTLAND
FT31
5 2000
19 SCOTLAND
FT31
6 2000
19 SCOTLAND
FT31
7 2000
19 NORTHERN IRELAND
FT31
8 2000
19 NORTHERN IRELAND
FT31
9 2000
19 NORTHERN IRELAND
FT31
10 2000
19 NORTHERN IRELAND
FT31
11 2000
19 NORTHERN IRELAND
FT31
12 2000
19 GREATER NOTTINGHAM
FT31
13 2000
19 GREATER NOTTINGHAM
FT31
14 2000
19 GREATER NOTTINGHAM
FT31
15 2000
19 GREATER NOTTINGHAM
FT31
16 2000
19 GREATER NOTTINGHAM
FT31
17 2000
19 SOUTH MIDLANDS
FT31
18 2000
19 SOUTH MIDLANDS
FT31
19 2000
19 SOUTH MIDLANDS
FT31
21 2000
19 SOUTH MIDLANDS
FT31
22 2000
19 SOUTHERN
FT31
23 2000
19 SOUTHERN
FT31
24 2000
19 SOUTHERN
FT31
25 2000
19 SOUTHERN
FT31
26 2000
19 SOUTHERN
Category
YTD
TW Bud
TW L Yr
YTD Bud
YTD
FOOD ONGOING
61073410
3053670
2964728
58019739
FOOD NEW
3718571
189842
184313
3682951
NON FOOD ONGOING
10382479
497266
482782
9050243
NON FOOD NEW
632157
31607
30687
600549
CLOSED
1425542
70526
68472
1325904
FOOD ONGOING
8482794
464321
450797
9657884
FOOD NEW
557120
29615
28752
598230
NON FOOD ONGOING
1453547
74972
72788
1469459
NON FOOD NEW
0
0
0
0
CLOSED
199576
10924
10605
227222
FOOD ONGOING
40132252
2375953
2306751
38490453
FOOD NEW
2999054
179352
174128
3048991
NON FOOD ONGOING
7040406
408523
396624
6863190
NON FOOD NEW
971905
53072
51526
923461
CLOSED
988733
59430
57699
1057856
FOOD ONGOING
18381251
1035154
1005004
22152310
FOOD NEW
0
0
0
0
NON FOOD ONGOING
2387970
130709
126902
2718767
CLOSED
1025199
58274
56577
1258728
FOOD ONGOING
51956875
3154725
3062839
52999380
FOOD NEW
3818371
213523
207304
3800722
NON FOOD ONGOING
10742075
607544
589848
10206745
NON FOOD NEW
1015257
55218
53610
894545
CLOSED
817911
48752
47332
887297
Data Sheet – Normal View
66
L Yr
56329844
3575680
8786644
583057
1287286
9376586
580806
1426659
0
220604
37369372
2960185
6663291
896564
1027044
21507097
0
2639579
1222066
51455708
3690021
9909462
868490
861453
Date Sheet – Formula View
67
Report Sheet – Normal View
68
Report Sheet – Formula View
69
Appendix C – The Spreadsheet Auditing Software Test Form
General Details
Name:
Producer:
Other Details:
Overview
Where does the auditing software fit into the spreadsheet?
At what stage is the auditing software designed for use?
In general terms, what does the auditing software do?
Who is the auditing software aimed at?
70
Test Results
Error
No.
Type
1
Q
2
Q
3
Q
4
Q
5
Q
6
Q
7
Q
8
M
9
M
10
M
11
L
12
L
13
L
14
L
15
L
16
O
17
O
Total Passed:
Results
Total Half-Passed:
71
Total Failed:
Additional Features
What additional features does the software provide and how useful are they?
Problems
Are there any problems/potential problems with the software?
Overall Impression
What is your overall impression of the software?
72
Appendix D – Test Results and Screenshots
The following pages contain the Spreadsheet Auditing Software Test Forms for each
of the tests carried out. They also contain a selection of screenshots taken from each
test.
73
Spreadsheet Auditing Software Test Form
General Details
Name: Excel’s Built In Auditing Functions
Producer: Microsoft
Other Details:
Overview
Where does the auditing software fit into the spreadsheet?
The auditing software is built in to Excel 2000, and is therefore fully integrated within
Excel.
At what stage is the auditing software designed for use?
During development – presumably before implementation as it does not work on
protected worksheets.
In general terms, what does the auditing software do?
The built in software is essentially based around investigating the contents of an
individual cell. The software will point the user at the constituent cells making up a
formula and it is then down to the user to check these cells. It is pretty much a cell-bycell inspection tool.
Who is the auditing software aimed at?
The software appears to be aimed, primarily at the developer to allow model
debugging before release to users.
74
Test Results
Error
No.
Type
1
Q
2
Q
3
Q
4
Q
5
Q
6
Q
7
Q
8
M
9
M
10
M
11
L
12
L
13
L
14
L
Results
PASSED
The cell would not show any precedents as it contains a value.
In Formula view mode, this error was even more obvious.
PASSED
The cell would not show any precedents as it contains a value.
In Formula view mode, this error was even more obvious.
PASSED
The cell would not show any precedents as it contains a value.
In Formula view mode, this error was even more obvious.
PASSED
The cell would not show any precedents as it contains a value.
In Formula view mode, this error was even more obvious.
FAILED
Although cell examination showed that a value was being used, there was no
suggestion that this was incorrect.
FAILED
FAILED
Protection had to be turned off before any of the auditing functions could be
used – there is no option to highlight protected/unprotected cells.
ALMOST-PASSED
The software clearly demonstrated that row 51 was omitted from the
calculation when inspecting this cell
ALMOST-FAILED
Whilst the software clearly showed the constituents of the calculation, there
was no suggestion that these constituents were incorrect.
ALMOST-PASSED
The precedents were clearly shown to exclude the relevant cell upon individual
cell inspection
ALMOST-FAILED
The precedents clearly showed the cells that were being referenced but there
was no indication that these were incorrect.
ALMOST-FAILED
Whilst the software clearly showed which figure was being used, there was no
suggestion that this figure was not the figure required.
ALMOST-FAILED
Whilst the software clearly showed that all the values were totalled, it was not
obvious that this was incorrect.
ALMOST-PASSED
Although when viewing the erroneous cell alone the calculation could be
correct, when viewed alongside other cells in the same column, the error
becomes noticeable.
75
15
L
16
O
ALMOST-PASSED
Although when viewing the erroneous cell alone the calculation could be
correct, when viewed alongside other cells in the same column, the error
becomes noticeable.
FAILED
17
O
FAILED
Total Passed: 4
Total Almost-Failed: 4
Total Almost-Passed: 4
Total Failed: 5
76
Additional Fe atures
What additional features does the software provide and how useful are they?
As the software itself is Excel, it is difficult to decide what functions should be
classed as auditing functions and what should be classed as standard Excel functions.
The main feature provided by Excel for auditing that falls outside of the functions
included in the auditing toolbar, is the option to view the worksheet as formulas rather
than values. This proved very useful in comparing formulas with other formulas in the
same row/column to find inconsistencies, and in the identification of formulas that
had been replaced by values.
Problems
Are there any problems/potential problems with the software?
The main problem with the built in Excel functions lie with how they are used. The
fact that the user is investigating one cell at a time means that the method is almost a
cell by cell inspection process, which, although successful in diagnosing spreadsheet
problems, take a great deal of time and user analysis to perform. A major aim of
spreadsheet auditing software is to reduce the time spent on spreadsheet auditing and
to highlight potential errors rather than the whole spreadsheet. A further problem in
the use of the built in software tools is that when a cell refers to a formula on another
worksheet, the user is requested to follow the link to the other worksheet, which can
make it difficult to follow a long formula and to check constituent parts from another
worksheet.
Overall Impression
What is your overall impression of the software?
The software auditing tools provided with Excel do exactly what you would expect
them to do. They allow the user to examine a single cell and review the formula
therein. In addition to the identification of precedents and descendants of certain cells,
the software has a built in error function which identifies the rogue factor(s) in a
formula (assuming the developer has not suppressed errors using the ISERR function,
which basically allows the formula to replace an error return with a pre-defined value
such as zero). However, the major fault with this software is that it does not allow
analysis of a range of cells (although it is possible to leave the precedent/descendent
arrows highlighted for other cells this quite often causes more confusion than it helps
the user). This will in effect mean that when a full cell by cell audit does not take
place, the user will be, at least to a certain degree, using his own judgement to decide
which cells need investigation.
The software overall does not provide the user with suggestions of possible errors,
rather it provides a tool for investigating any cells identified as potentially unsafe by
the user. The reasonable results found in the test (4 passed, 4 almost-passed) are
largely down to the almost cell by cell inspection techniques necessary to use the
tools. It is obvious, yet still worth noting, that had the erroneous cells not been
subjected to the tools at the request of the tester, then the errors would have been
overlooked.
77
The fact that the worksheet had to be unprotected before cells could be audited means
that in most cases, the use of these tools would be restricted to the developer of the
spreadsheet, meaning that once the spreadsheet model is ‘out in the field’, the auditing
tools would be unusable.
It is essential to remember that the software tools tested here are provided free with
Excel, and as such do not purport to do as thorough audit as those provided, usually at
an extra cost, by a third party.
78
Excels built in auditing functions successfully find the error in cell D27
79
Errors in cells D27 and E13 are easily spotted in formula view
80
The error in cell P44 is only noticeable when viewed alongside other cells in the same column.
81
Spreadsheet Auditing Software Test Form
General Details
Name: The Spreadsheet Detective
Producer: Southern Cross Software Queensland
Other Details: Shareware/Licenceware – Approximately £100 for the full registered
version.
Overview
Where does the auditing software fit into the spreads heet?
The Spreadsheet Detective software is, in effect an Excel Add-in. Once installed, an
extra menu is added to Excel and an extra toolbar can be added to access the features
available.
At what stage is the auditing software designed for use?
The Spreadsheet Detective requires that all protected worksheets and workbooks
should be unprotected so it appears that the software is primarily designed for the
developer to use in pre-production testing.
In general terms, what does the auditing software do?
The Spreadsheet Detective basically analyses a spreadsheet to identify sets of
formulas (referred to as schemas), simplify the reading of these schemas, and
identifies potential trouble spots for further analysis. Various reports can be printed
enabling the auditor to view and work on spreadsheet errors on paper.
Who is the auditing software aimed at?
The software is aimed largely at the developer of the spreadsheet to identify
errors/potential errors within the spreadsheet.
Test Results
82
Error
No.
Type
1
Q
2
Q
3
Q
4
Q
5
Q
6
Q
7
Q
8
M
9
M
10
M
11
L
12
L
13
L
14
L
Results
PASSED
In both the shaded sheet and the annotated sheet view the value rather than a
formula in this cell stood out.
PASSED
In both the shaded sheet and the annotated sheet view the value rather than a
formula in this cell stood out.
PASSED
In both the shaded sheet and the annotated sheet view the value rather than a
formula in this cell stood out.
PASSED
In both the shaded sheet and the annotated sheet view the value rather than a
formula in this cell stood out.
PASSED
On the annotated sheet view, the software reports constants within formula and
the schema containing them.
ALMOST-FAILED
The presence of ‘new’ schema in the adjacent cells could highlight this error.
PASSED
At the foot of the annotated wor ksheet, the software includes a list of
unprotected schemas. As this should be limited to just 5 schemas on the input
sheet, errors are easy to spot.
ALMOST-PASSED
Whilst the annotation of the cell clearly showed the new formula, the lack of
either a clear outline of the constituent cells or a note in the footer report
means that this could be missed
PASSED
The automatic naming function of the software made spotting this error
particularly easy – this function means that the formulas almost read as
English!
ALMOST-PASSED
Whilst the annotation of the cell clearly showed the new formula, the lack of
either a clear outline of the constituent cells or a note in the footer report
means that this could be missed.
PASSED
The automatic naming function of the software made spotting this error
particularly easy.
PASSED
The automatic naming function of the software made spotting this error
particularly easy.
PASSED
The software indicates the presence of a possible error in the erroneous cell by
listing it under the heading ‘Reference to Non Numerics’ in the footer report.
PASSED
The software spots the presence of a new schema when it hits this cell as the
calculation is different from the previous schema. The annotation then makes
this error quite easy to spot.
83
15
L
16
O
PASSED
The software spots the presence of a new schema when it hits this cell as the
calculation is different from the previous schema. The annotation then makes
this error quite easy to spot.
FAILED
17
O
FAILED
Total Passed: 12
Total Almost-Failed: 0
Total Almost-Passed: 2
Total Failed: 3
84
Additional Features
What additional features does the software provide and how useful are they?
The software provides many additional features that, whilst not of particular
relevance to the test itself, appear to be useful functions for the spreadsheet model
developer. These include the following:
Formula Report: This presents all formulas from a selected worksheet/workbook on
an easy-to-read/audit report.
Inter-worksheet Report: This gives an indication of how the data flows between
worksheets
Workbook Precedent Report: This presents a report of all external links to other
workbook files.
Sensitivity Report: This report attempts to highlight sensitivity of ‘bottom line’
figures n a calculation to a change n a particular constituent cell.
A function to compare two versions of a worksheet that have been worked upon
separately.
Problems
Are there any problems/potential problems with the software?
The different types of shading, which try and identify which formulas have been
copied and from where, can be very confusing, particularly where schemas overlap.
However this problem has been partially resolved by providing an option to replace
shading with an identification character on each cell, but this can still be somewhat
confusing. I found that the annotation sheet was easier to use (despite it’s apparent
complexity), for all but the most basic of errors.
85
Overall Impression
What is your overall impression of the software?
The software performed particularly well in the tests, only really falling over on the
errors that were particularly difficult to spot. The fact that it can work on a whole
workbook at a time means that it is much faster than the built ni Excel auditing
functions. The footer report produced when annotating a worksheet proved
particularly useful in error spotting, particularly the more subtle errors such as
unprotected ranges and totals that included sub totals (in addition to data rows). The
automatic naming worked well in the annotation of formula and saved a lot of
potentially erroneous dashing around the spreadsheet to match labels to data. The
identification of schemas was well done (at least on the annotation view) and enable
the manual audit of a much more limited number of cells than the cell-by-cell
approach necessary for confident auditing with Excel. The Formula Report and the
Cell Precedent Report are very easy to read and the fact that these were produced in a
separate workbook meant they could be stored and checked later (although to take
advantage of the ‘jump to’ options you do, of course need the original workbook
open). However, it must be remembered that the software is still just a tool, and
despite it providing a list of potential problem cells in the footer report of the
annotated worksheet view, many of the errors are still within the schemas identified
within the sheet itself, and the auditor will still have to manually audit each schema’s
formula. The software also missed the errors that would be much more visible to the
human eye such as the omission of rows between the input sheet and the report sheet.
86
The Spreadsheet Detective makes the lack of a formula in cell D27 obvious using either the shaded formula view or the annotated worksheet
view
87
The error in cell O59 is easily spotted thanks to the Spreadsheet Detective’s automatic allocation of names to cells during the auditing process.
i.e. “NonFoodNew + NonFoodNew + FoodNew”
88
The Spreadsheet Detective allows the user to view all the precedents for a particular cell on a one-page report.
89
Spreadsheet Auditing Software Test Form
General Details
Name: The Excel Auditor
Producer: Byg Software
Other Details: Costs £99
Overview
Where does the auditing software fit into the spreadsheet?
The Excel Auditor is an add-in for Excel. It provides functions across a number of
different fields, including spreadsheet auditing.
At what stage is the auditing software designed for use?
The Excel Auditor, in addition to providing auditing functions, provides functions for
various development aids such as the Object Properties Modifier. It also provides
tools for all users that simplify and expand upon functions that already exist within
Excel such as the zooming and search functions. As many functions will work even
on protected worksheets and the functions are spread across many areas, it appears
that the software is designed for use at all stages of the spreadsheet’s life.
In general terms, what does the auditing software do?
Although the functionality of The Excel Auditor stretches beyond the provision of
auditing tools, the auditing tools are those under evaluation. The software provides
two primary and two secondary audit tools. The primary tools are the Audit Map and
the Worksheet Cell Analyser. The secondary tools are the Cell Roots and Circular
References Analyst functions. Each of these tools will analyse a worksheet or a
particular cell and produce a report of its findings on a separate workbook.
Who is the auditing software aimed at?
The Excel Auditor is, in effect, a collection of individual tools. These tools are aimed
at all users, from the developer in the case of the Object Properties Modifier, to the
user, in the case of the zoom and search tools, through to the auditor, in the case of the
audit tools.
90
Test Results
Error
No.
Type
1
Q
2
Q
3
Q
4
Q
5
Q
6
Q
7
Q
8
M
9
M
10
M
11
L
12
L
Results
ALMOST-FAILED
Whilst the Worksheet Analyser and audit map clearly shows which cells on the
worksheet have formulas, and the contents of those formula, the results are on
a new sheet so errors are difficult to spot.
ALMOST-FAILED
Whilst the Worksheet Analyser and audit map clearly shows which cells on the
worksheet have formulas, and the contents of those formula, the results are on
a new sheet so errors are difficult to spot.
ALMOST-FAILED
Whilst the Worksheet Analyser and audit map clearly shows which cells on the
worksheet have formulas, and the contents of those formula, the results are on
a new sheet so errors are difficult to spot.
ALMOST-FAILED
Whilst the Worksheet Analyser and audit map clearly shows which cells on the
worksheet have formulas, and the contents of those formula, the results are on
a new sheet so errors are difficult to spot.
ALMOST-FAILED
Whilst the Worksheet Analyser clearly showed that a fixed value was being
used, there was no suggestion that this was incorrect.
FAILED
FAILED
The DocIt function, that documents workbooks, merely shows which sheets
are protected, and does not specify which cells are left unpr otected within
those sheets.
ALMOST-FAILED
Both the Worksheet Analyser and the Cell Roots function clearly show the
constituents of the formula, but as these are shown on a separate report, it is
difficult to spot the error.
ALMOST-FAILED
Both the Worksheet Analyser and the Cell Roots function clearly show the
constituents of the formula, but as these are shown on a separate report, it is
difficult to spot the error.
ALMOST-FAILED
Both the Worksheet Analyser and the Cell Roots function clearly show the
constituents of the formula, but as these are shown on a separate report, it is
difficult to spot the error.
ALMOST-FAILED
The software clearly shows the constituents of the formula, but fails to make
the error obvious. The use of some sort of constituent naming scheme would
be useful here.
ALMOST-FAILED
The software clearly shows the constituents of the formula, but fails to make
the error obvious. The use of some sort of constituent naming scheme would
91
be useful here.
13
L
14
L
15
L
16
O
ALMOST-FAILED
Both the Worksheet Analyser and the Cell Roots function clearly show the
constituents of the formula, but as these are shown on a separate report, it is
difficult to spot the error.
ALMOST-FAILED
The software clearly shows the constituents of the formula, but fails to make
the error obvious. When viewed alongside other cells this could be more
obvious, but as the Worksheet Analysis report for the Report sheet was 64
pages long, this error was still well hidden.
ALMOST-PASSED
The software clearly shows the constituents of the formula, but fails to make
the error obvious. When viewed alongside other cells this could be more
obvious, but again, in the Worksheet Analysis report for the Report sheet, this
error was still well hidden.
FAILED
17
O
FAILED
Total Passed: 0
Total Almost-Failed: 12
Total Almost-Passed: 1
Total Failed:4
Additional Features
What additional features does the software provide and how useful are they?
The software provides various tools beyond the auditing functions evaluated. These
tools tend to cover functions that are already available in Excel, but tend to make
them more powerful and user-friendly than their original counterparts. The software
also provides tools that allow the user to build up a list of items such as named ranges,
modules, charts and worksheets and provides summary statistics for use in
spreadsheet documentation.
Problems
Are there any problems/potential problems with the software?
There are a number of problems with the software and it’s use in auditing
spreadsheets. The major problem is that the software only produces reports on a
separate workbook rather than providing some graphical or visual representation of
the formula on the worksheet itself. This means that errors such as missing a row from
a SUM calculation, which would be obvious with some form of visual representation
on the worksheet are less obvious to the auditor. The software also failed to identify
any pattern or schema to the formulae and as such had to produce a full list of formula
in the reports, which on the Report sheet of the test spreadsheet resulted in a 64 page
report. The software would benefit from its use alongside the built-in auditing
92
functions in Excel, such as the cell precedents and dependants (although the software
will provide it’s own report of these precedents/dependants), yet the software seemed
to have the effect of disabling some of these built-in functions. The software also
failed to identify any potential error cells (such as those with values in formulae) and
had no way of identifying formulas that had been overtyped with values.
Overall Impression
What is your overall impression of the software?
Although, despite its name, The Excel Auditor includes functions outside the us ual
scope of auditing software, the tools included to assist in the audit of spreadsheets are
rather limited. Most of the auditing functions involve the production of a separate
report which, although technically correct, need to be viewed alongside the or iginal
worksheet(s) to audit the cells. The software does not attempt to identify any patterns
or schema in the formula and as a result the reports become an assistant for cell by
cell inspection as a means of auditing. The lack of any overlay or visual
representation of the worksheets means that errors involving formulas that have been
overtyped with values or incorrectly specified summed ranges much more difficult to
spot. The audit tools appear to be more suited to the documentation of a spreadsheet
rather than the auditing of a spreadsheet with the Worksheet Cell Analyser and the
Lister tools allowing the formulae and constituents of a worksheet to be documented
with minimum user intervention on a report. The Excel Auditor is likely to be of most
use as a report producing tool that assists in the documentation of spreadsheets or
auditing on a cell by cell basis when the auditor is not connected to the PC. The
software does not attempt to highlight any particular type of error, rather it will
document a whole worksheet (or, in some cases, a single cell) and then make the
formula easier to audit by the auditor. Perhaps surprisingly, The Excel Auditor is
more useful as a documentation and development tool, than it is an auditing tool. The
Excel Auditor is, however, very easy to use and basic changes to allow some form of
visual overlay to the original spreadsheets, and the identification of schema, could
significantly increase it’s power as an auditing tool. These issues meant that on the
test spreadsheet, many errors that could have achieved a PASSED result, merely
achieved an ALMOST-PASSED or ALMOST-FAILED result, largely down to the
fact that the errors where more difficult to spot from a separate worksheet.
93
The Excel Auditor produces a traditional audit map of a worksheet – in this case the Input sheet.
94
The Excel Auditor produces a breakdown of formulas on a worksheet
95
The Excel Auditor documentation function gives general data about the workbook.
96
Spreadsheet Auditing Software Test Form
General Details
Name: Operis Analysis Kit
Producer: Operis Business Engineering Limited
Other Details:
Overview
Where does the auditing software fit into the spreadsheet?
The Operis Analysis Kit is an add-on for Excel. Once installed the functions are
availa ble via an extra menu.
At what stage is the auditing software designed for use?
The fact that worksheet protection needs to be disabled before many of the functions
can be used, coupled with extra functions such as the add version function and the
extra naming functions suggest that the Operis Analysis Kit is primarily a used in the
development and maintenance of spreadsheet models.
In general terms, what does the auditing software do?
The software, from an auditing viewpoint, has two main aspects. The first allows the
software user to search the worksheet for particular types of cell, such as those with
no dependants or those that include formulas with hardwired constants. The second
aspect concentrates on graphically mapping the cells on a worksheet, to identify
formula schema (referred to as distinct formula) and types of cell,
(label/formula/number etc.) and to document the worksheets distinct formulae and
relevant statistics.
Who is the auditing software aimed at?
As any worksheet protection needs disabling, the auditing software seems to be aimed
primarily at spreadsheet developers and auditors.
97
Test Results
Error
No.
Type
1
Q
2
Q
3
Q
4
Q
5
Q
6
Q
7
Q
8
M
9
M
10
M
11
L
12
L
Results
PASSED
The option to overlay colour-coded cells on to the original spreadsheet, or into
a new spreadsheet with the data copied in, means that this error is relatively
obvious.
PASSED
The colour coding of cells makes this error particularly obvious on the Report
worksheet as this is the output sheet.
PASSED
The colour coding of cells makes this error particularly obvious on the Report
worksheet as this is the output sheet.
PASSED
The colour coding of cells makes this error particularly obvious on the Report
worksheet as this is the output sheet.
PASSED
The search for hardwired constants function correctly identified the whole of
column K as potential errors.
ALMOST-FAILED
Although when the whole worksheet is mapped, the adjacent formulae are
highlighted because they are not ‘in sync’ with the one above, the error is not
highlighted on the label cell.
FAILED
The worksheets need to be unprotected before OAK can be used. There is no
attempt to identify ‘open’ cells.
ALMOST-PASSED
Whilst the formula was marked as distinct and individual formula inspection
using the summary report would highlight the formula, there is no suggestion
that this formula is incorrect.
ALMOST-PASSED
Whilst the formula was marked as distinct and individual formula inspection
using the summary report would highlight the formula, there is no suggestion
that this formula is incorrect.
ALMOST-PASSED
Whilst the formula was marked as distinct and individual formula inspection
using the summary report would highlight the formula, there is no suggestion
that this formula is incorrect.
ALMOST-FAILED
Whilst the formula was marked as distinct and individual formula inspection
using the summary report would highlight the formula, there is no suggestion
that this formula is incorrect.
ALMOST-FAILED
Whilst the formula was marked as distinct and individual formula inspection
using the summary report would highlight the formula, there is no suggestion
that this formula is incorrect.
98
13
L
14
L
15
L
16
O
PASSED
Using the search – references to blank cells option the rogue cell was
identified.
FAILED
The formula was not highlighted as distinct as it was a copy of the formula to
the left, rather than the formula above as it should have been.
PASSED
The formula was correctly highlighted as distinct.
FAILED
There are no functions to identify text patterns and the formula are calssed as
unique all the way down the page due to the adjacent cell limitation
17
O
PASSED
Using the search – un-referenced cells options on the FILE sheet highlighted
the rogue data row.
Total Passed: 8
Total Almost-Passed: 3
Total Almost-Failed: 3
Total Failed: 3
Additional Features
What additional features does the software provide and how useful are they?
In addition to the auditing features, the Operis Analysis Kit includes tools to
manipulate Excel named ranges by emulating and expanding the naming functions
that are standard in Excel. The software also includes an add version function which
automates the version control documentation ensuring that all relevant details are
recorded in a standard format on the first worksheet of the model.
Problems
Are there any problems/potential problems with the software?
The Operis Analysis Kit is limited slightly in its ability to identify distinct formulae in
that it relies upon the formula logically matching the formula in an adjacent cell. This
could lead to duplication of work in the individual inspection of formulae.
99
Overall Impression
What is your overall impression of the software?
The software runs from an additional short menu within Excel, and as a result looks,
at first glance to be slightly simplistic. However, the menu is particularly well laid out
and, from an auditing viewpoint, can almost be used as a checklist of the auditing
tools available. The deceptively simple search sub-menu gives rise to a series of
powerful tools that proved quite adept at spotting the errors in the test. Whilst the
distinct formulae identification was slightly flawed, it did at least err on the side of
caution by identifying too many rather than too few distinct formulae in most cases,
which means that the subsequent cell by cell inspection would at least find any errors
in these cells. The search for cells without dependants even highlighted the extra data
held in the imported file that did not feed through to the Report sheet. In the test the
Operis Analysis Kit either identified or hinted at most of the more ‘findable’ errors,
and with the exception of it’s inability to identify unprotected cells, only really
struggled on those errors which could be considered more likely to be spotted by the
human eye than a program following a set of pre-defined rules.
100
Operis Analysis Kit makes spotting the error in cell D!27 easy thanks to the yellow shading for values and the sky blue shading for formulae.
101
The Summarize Workbook command provides the report above, which attempts to identify distinct formulae in addition to trying to quantify the
complexity of formulae.
102
The un-referenced cells on the File worksheet clearly show which cells are not being carried through to the Report worksheet.
103
Spreadsheet Auditing Software Test Form
General Details
Name: SpACE (Spreadsheet A uditing from C ustoms and Excise)
Producer: Customs and Excise In-house software
Other Details:
Overview
Where does the auditing software fit into the spreadsheet?
SpACE works on a copy of a workbook. The workbook is specified at the outset and
SpACE will then copy each worksheet, using a ‘brute-force’ approach to remove any
passwords, into a new workbook.
At what stage is the auditing software designed for use?
SpACE is an in -house system designed and used by HM Customs and Excise
inspectors in the examination and audit of spreadsheets used to calculate a company’s
tax liability. As such it was originally a post-implementation audit tool. More recently
Customs and Excise have been marketing the software as a general auditing tool to
private businesses to allow it’s use in general spreadsheet auditing.
In general terms, what does the auditing software do?
The software, working on an unpr otected copy of the spreadsheet, using a
combination of search facilities, overlaid mapping options, and identification of
unique formula, attempts to highlight potential errors in a spreadsheet.
Who is the auditing software aimed at?
SpACE was originally written as a tool for Customs and Excise to use whilst auditing
spreadsheets used to calculate tax liability. As the software has evolved, it has been
made available as a general spreadsheet auditing tool.
104
Test Results
Error
No.
Type
Results
1
Q
PASSED
Using a colour coded overlay the software clearly identifies the data type in
each cell, which, coupled with the identification of cells with no dependants
meant that this error is obvious. This cell is also identified in the summary
report as a numeric cell with no dependants.
PASSED
Using the colour coded overlay the numeric value stands out on the map.
2
Q
3
Q
PASSED
Using the colour coded overlay the numeric value stands out on the map.
4
Q
PASSED
Using the colour coded overlay the numeric value stands out on the map.
5
Q
6
Q
7
Q
8
M
9
M
10
M
11
L
12
L
ALMOST-PASSED
The software allows the use to search for cells containing a particular value
(Within user -definable tolerance limits). Once a value was identified (in this
case 2.5% for inflation), the software highlights every cell that contains that
value.
ALMOST-PASSED
The function to search for groups of text strings, along with the indication that
the adjacent formulae are unique as they go out of sync with the others, allow
the user to visually spot label patter ns, although cause of the problem is not
instantly apparent.
PASSED
Although the software removes the sheet protection (even if the sheet is
protected by passwords), the user can choose to map all unprotected cells on a
worksheet which makes this error stand out.
ALMOST-PASSED
The software clearly identifies the offending cell as a new unique formula, so
when the cell is examined the error should be spotted.
ALMOST-PASSED
The software clearly identifies the offending cell as a new unique formula, so
when the cell is examined the error should be spotted.
ALMOST-PASSED
The software clearly identifies the offending cell as a new unique formula, so
when the cell is examined the error should be spotted.
ALMOST-PASSED
Using the options to link the root and bottom line cells, the software failed to
establish a link. Further investigation revealed the lack of a connection was
down to the input values not being reflected in this bottom line figure.
ALMOST-PASSED
Using the options to link the root and bottom line cells, the software failed to
establish a link. Further investigation revealed the lack of a connection was
down to the input values not being reflected in this bottom line figure.
105
13
L
PASSED
The software clearly identifies the offending cell as a new unique formula, so
when the cell is examined the error should be spotted.
14
L
PASSED
Whilst the software did not identify the offending cell as a unique formula, it
was obvious that the formula matched that to the left rather than the above,
which was the case in the rest of the column. The identification of the source
formula cell on the map made this even more obvious.
15
L
PASSED
The software clearly identifies the offending cell as a new unique formula, so
when the cell is examined the error should be spotted.
16
O
ALMOST-PASSED
The software identified a new formula scheme for row 44 of the Report sheet,
rather than using the scheme identified in row 18. This was due to the fact that
4 rows were added rather than 5. This, coupled with the options available to
highlight text strings with different coloured backgrounds should enable the
missing line to be identified.
17
O
PASSED
The ‘cells with no dependencies’ identification on the File sheet, identified a
whole row of cells with no dependencies, which meant that this error was easy
to spot.
Total Passed: 9
Total Almost-Passed: 8
Total Almost-Failed: 0
Total Failed: 0
106
Additional Features
What additional features does the software provide and how useful are they?
In addition to the more traditional auditing functions incorporated in SpACE, there
are functions to identify whether automatic calculation is enabled, work on all
worksheets in a workbook at once, check lists of numbers for duplicates, attempt to
identify the numbers that make up a particular total, compare different versions of
worksheets, hide particular worksheets that have been added by SpACE and search
for particular values such as VAT rates and VAT numbers.
Problems
Are there any problems/potential problems with the software?
There is no doubt that SpACE is a particularly powerful auditing tool. Perhaps the
only drawback could be the absence of any form of cell labelling based upon row and
column headings. It could also be argued that an inexperienced user may struggle to
understand the in-depth colour schemes that are included in the audit maps, however,
this should not prove a long term problem.
Overall Impression
What is your overall impression of the software?
It is apparent that SpACE has been in ‘active service’ for some time and used
extensively throughout that time. The software identified every error in the test,
usually in a particularly obvious way including those errors which could be described
as more obvious to the human eye. The software’s power and complexity are reflected
in the fact that users can undergo a two day course before they begin to use the
software. The identification of unique formulae worked particularly well with very
few formulae being incorrectly identified as unique. Functions such as the
identification of cells with no dependants allowed the identification of un-used data
from the imported file and the route finder function identified the break in the chain
between the logical root and bottom-line cell highlighting the error caused by using
the wrong year in the ‘Revalued last year’ column. The colour coding, whilst initially
quite confusing, was particularly useful in the identification of errors involving unreferenced cells and numbers that had overwritten formulae.
107
The lack of a formula in cell D7 is obvious thanks to the cell shading (green = numeric) and the lack of dependant cells (red border).
108
Although SpACE removes worksheet protection before use, it can identify which cells are set as unprotected.
109
SpACE provides an overview of all the worksheets in a workbook, allowing the auditor to estimate the complexity and subsequently the risks
involved in the use of the spreadsheet.
110
The colour coded map overlay used to identify unique formulae, clearly identifies formula schemes and includes the original formula cells
reference on the map.
111
Appendix E – Project Proposal
Introduction
This document proposes that an investigation takes place into spreadsheet auditing,
particularly why it should be done, how it can be done and how successful it can be.
Background
The author has worked in a semi-IT role for the past 11½ years at the CWS (7½ years
in the Pensions department and 4 years in the Retail Accounting department). During
this time the author has developed a number of spreadsheet models for both his own
use, and use by colleagues. As errors have been found in these spreadsheets, the
author is interested in investigating how these errors occur and how they can be found
and corrected. After some initial research into this field (see bibliography for details),
the author proposed the following 5 options for further investigation:
1. Continue the investigation into spreadsheet errors started by Paul Creely in his
project last year.
2. Start with a summary of the current thinking on types and causes of spreadsheet
errors and attempt to design a framework/methodology to be used when
developing spreadsheets that aims to reduce these errors.
3. From a basis of the current thinking on spreadsheet errors, look at devising a
training program/guide with standard/general policies for spreadsheet developers.
4. A review and critique of the methodologies/frameworks currently available/in use
in spreadsheet design.
5. An investigation into spreadsheet auditing i.e. why and how it is done and what
success does it have. This should include both manual auditing and software
auditing tools.
After consultation with Mike O’Hara (project supervisor) and Ray Butler (spreadsheet
guru), and a further review of the research done the author decided to progress with
the final option put forward, that of spreadsheet auditing.
112
Project Definition
The Current Situation
A significant amount of research has been published on both frequency and type of
spreadsheet errors. However, the research into how developers detect and correct
spreadsheet errors is somewhat more limited. There are also software tools available
that aim to assist spreadsheet developers in the detection of these errors, although the
author was unable to find any evidence published that demonstrated the impact of
these tools on error detection.
Project Aims and Objectives
The project has the following primary objectives:
•
•
•
•
•
Investigate, summarise and analyse the research to date on spreadsheet errors.
Investigate, summarise and analyse research into how developers try to avoid
software errors from occurring.
Investigate, summarise and analyse how spreadsheets are audited.
Identify and review the software available to assist in spreadsheet auditing.
Apply the spreadsheet auditing methods and software tools identified to either a
standard custom built spreadsheet, or a series of spreadsheet models to investigate
their impact on error detection/correction.
The secondary objective is as follows:
•
To test the different methods/software tools available against a standard
spreadsheet in an experimental environment with a number of different subjects.
Resources
The author intends to use the following resources to achieve the objectives identified:
1. Published academic reports and research from traditional sources such as libraries
and publications and modern sources such as the Internet.
2. The author is in communication with Mike O’Hara and Ray Butler and intends to
take advantage of their expertise in this field.
3. The author will use both Microsoft Excel and Lotus 1-2-3 for testing and research
purposes, along with Microsoft Office, Lotus SmartSuite and Microsoft Project
for general project administration purposes.
4. The author also intends to attend a lecture on spreadsheets given by Ray Butler on
9th November 2000 entitled ‘the eXcel files: The trouble with spreadsheets’.
113
Project Risks
A major risk with any project of a reasonable size is that of achieving deadlines. To
achieve these deadlines the author has developed a Gantt chart to plan the project and
has included extra time for those tasks which may prove more difficult and timeconsuming than first thought. This Gantt chart has been developed to plan the project
will inevitably need to be flexible enough to deal with changes that occur during the
life of the project. It will therefore be interesting to compare the initial Gantt chart
with the final Gantt chart and this will be a significant part of the academic review
included in the final report. A breakdown of the tasks identified for the project and
included in the Gantt chart is in the following section entitled ‘Project Planning’.
A risk that exists within the project plan in question, concerns the staging of the
experiment that will test the various spreadsheet auditing methods/software tools
identified using independent testers. As with any task that involves a third party a
certain risk is involved, however, the author will definitely review the methods and
software available on a personal level and should be able to draw adequate
conclusions from this review, should the experiment become impractical.
Another area that will involve third parties will be the acquisition of the spreadsheet
auditing software for review. With the assistance of Ray Butler and Mike O’Hara the
author should be able to acquire the software, hopefully at minimum expense, in time
to allow a full review.
Project Planning
The author identified the following tasks as essential in the completion of this project.
These tasks were then arrange d in a Gantt chart to show the time-scales and
dependencies involved, along with the deadlines set out in the project handbook.
•
•
•
•
•
•
•
Investigate/Write up spreadsheet history – Why spreadsheets are so popular, what
they are used for and how they have evolved?
Investigate/Write up spreadsheet error research – What research has taken place
into spreadsheet errors, what conclusions can be drawn and what types of errors
have been identified?
Investigate/Write up spreadsheet development techniques – What are developers
doing to prevent spreadsheet errors?
Investigate/Write up spreadsheet error impact – Why should we worry about
spreadsheet errors?
Investigate/Write up spreadsheet auditing research/methods – How are
spreadsheets audited, how successful is spreadsheet auditing?
Develop sample spreadsheet and scenario for testing – For use in both the authors
personal review of spreadsheet auditing tools and techniques, and also in the
proposed auditing experiment.
Identify/Acquire software auditing tools available – The tools identified as having
a possibility for further review include SPACE, currently in use in Customs and
Excise, The Excel Auditor, produced by Byg Software, and The Spreadsheet
114
•
•
•
•
•
•
•
•
•
•
Detective, produced by Southern Cross Software, along with the built -in auditing
tools provided by Microsoft within Excel 2000.
Plan auditing experiments – This will involve both the authors own review and the
proposed independent auditing experiment.
Do auditing experiment
Review/conclude experiments – What have the experiments shown, how do they
suggest that spreadsheet auditing should be done?
Write up reviews/conclusions
Conclude the project – Conclude the whole project, from the research performed
so far to the conclusions drawn from the authors experiments.
Academic review – An investigation into what has been learnt by the author, how
the project developed throughout it’s lifetime and whether the objectives set out at
the start of the project have been met.
Write up interim report – How is the project progressing, what changes have been
made to the plan since the project began and what problems have been
encountered?
Write up final report
Prepare presentation
Perform presentation
115
Gantt Chart
ID
1
2
3
4
5
6
7
8
9
Task
Submit the project proposal
Attend the spreadsheet lecture
Investigate spreadsheet history
Write up spreadsheet history
Investigate spreadsheet error research
Write up spreadsheet error research
Investigate spreadsheet development techniques
Write up spreadsheet development techniques
Investigate spreadsheet error impact
Duration
0 days
0 days
56 days
28 days
56 days
28 days
56 days
28 days
56 days
Start
Fr i 27/10/00
Thu 09/11/00
Fri 27/10/00
Mon 15/01/01
Fri 27/10/00
Mon 15/01/01
Fri 27/10/00
Mon 15/01/01
Fri 27/10/00
End
Fri 27/10/00
Thu 09/11/00
Fri 12/01/01
Wed 21/02/01
Fri 12/01/01
Wed 21/02/01
Fri 12/01/01
Wed 21/02/01
Fri 12/01/01
Dependencies
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
Write up spreadsheet error impact
Investigate spreadsheet auditing research/methods
Write up spreadsheet auditing research/methods
Write up the interim report
Submit the interim report
Develop sample spreadsheet and scenario for testing
Identify/acquire software auditing tools available
Plan auditing experiments
Do auditing experiment
Review/conclude experiments
Write up reviews/conclusions
Conclude the project
Academic Review
Write up the final report
Prepare presentation
28 days
56 days
28 days
14 days
0 days
21 days
49 days
14 days
21 days
14 days
21 days
14 days
14 days
21 days
7 days
Mon 15/01/01
Fri 27/10/00
Mon 15/01/01
Mon 15/01/01
Thu 01/02/01
Fri 27/10/00
Fri 27/10/00
Mon 27/11/00
Fri 15/12/00
Mon 15/01/01
Fri 02/02/01
Mon 05/03/01
Fri 23/03/01
Thu 22/02/01
Fri 23/03/01
Wed 21/02/01
Fri 12/01/01
Wed 21/02/01
Thu 01/02/01
Thu 01/02/01
Fri 24/11/00
Wed 03/01/01
Thu 14/12/00
Fri 12/01/01
Thu 01/02/01
Fri 02/03/01
Thu 22/03/01
Wed 11/04/01
Thu 22/03/01
Mon 02/04/01
9
1
11
0 days
0 days
Mon 02/04/01
Tue 01/05/01
Mon 02/04/01 24
Tue 01/05/01 23
25 Perform presentation
26 Submit the final report
116
1
1
3
1
5
1
7
1
13
1
1
15
17
18
19
4,6,8,10,12,20
21
4,6,8,10,12
23
Project Completion
The project will be deemed a success if all the objectives identified above are
attained. A qualified success could be attained if all of the primary objectives
ide ntified are achieved.
117
Appendix A
Project/Report Plan
Project to include....
Sections:
A - Introduction
B - Findings
C - Conclusions
X - Appendix
Section
Part
A
A
1
2
A
3
A
4
B
1
B
2
B
3
B
B
C
3a
3b
1
C
2
C
C
3
4
X
X
1
2
S/s history
Summary of research into errors on s/s
Summary of types of errors
What damage are caused by errors?
Why bother trying to stop errors?
How can we reduce s/s errors?
- what constitutes good s/s design?
-- good documentation/practice/standards etc.
Investigation of research into auditing s/s
- manual auditing
- s/w assisted auditing
- code inspection & s/w engineering tools/techniques
Summary of research into auditing spreadsheets
- how successful is it?
- what’s good/bad about it?
- etc.
What do s/w auditing assistants do?
Which are available?
Do a comparison/review of those available?
How do they work?
How successful are they?
Test s/w auditing programs on sample s/s
Practical experiment on sample s/s
Do s/w auditing programs help?
- if so how much?
How should we audit s/s ?
- accuracy/error free
- documentation
What next for s/s auditing research?
Academic review
- did I achieve objectives set out in proposal/plan?
- what did I learn?
- personal conclusions
Develop a sample s/s for testing on s/w auditors and possible practical experiment
Results from comparisons/experiments
118
Tasks & Deliverables
Tasks
Investigate s/s history (literature review)
Write up s/s history
Investigate s/s error research (literature review)
Write up s/s error research
Investigate s/s error impact (literature review)
Write up s/s error impact
Investigate s/s auditing research/methods (literature review)
Write up s/s auditing research/methods
Develop sample s/s & scenario for testing
Find out/get hold of what s/w auditing tools exist
Plan auditing experiment
Use & review s/w tools on sample s/s
Do auditing experiment
Review/conclude experiments
Write up reviews
Conclude all report
Write up conclusions/recommendations for further review etc.
Academic review
Write up academic review
Project deliverables
Project Proposal - w/e 20/10/00
Interim Report - 1/2/01
Final Report - 1/5/01
Presentation - w/c 8/5/00
119
Appendix F – Interim Report
Introduction
This report provides a review of the progress made on the objectives stated in the
Project Proposal. It contains details of the work done, and changes in the project plan
outlined in the Project Proposal. It aims to document any change in emphasis and
direction of the project and any relevant issues that have arisen in the first part of the
project.
Objectives
The following objectives were stated in the Project Proposal. A review of progress on
those objectives, and changes made to those objectives, follows.
Primary Objectives
Ø Investigate, summarise and analyse the research to date on spreadsheet errors.
Ø Investigate, summarise and analyse research into how developers try to avoid
software errors from occurring.
Ø Investigate, summarise and analyse how spreadsheets are audited.
Each of these three objectives has, in the main part, been achieved, although if time
permits further research, analysis and documentation may be added. A sample of the
work produced in the achievement of these objectives is included in the appendix to
this report. Whilst undertaking this work it became apparent that the authors initial
impression of the situation, resulting from preliminary research, that although
extensive research had taken place into investigating and identifying errors and
developing error avoidance techniques, research into auditing spreadsheets was
limited, and that into software auditing tools was scarce, was correct.
Ø Identify and review the software available to assist in spreadsheet auditing.
Ø Apply the spreadsheet auditing methods and software tools identified to either a
standard custom built spreadsheet, or a series of spreadsheet models to investigate
their impact on error detection/correction.
During the course of the research, the following spreadsheet auditing assistants have
been identified:
•
•
•
•
•
•
Excel’s built-in auditing tools
Operis Analysis Kit (add-in)
The Spreadsheet Detective
The Excel Auditor
Space
The Spreadsheet Professional
120
It is intended to select three or four of these products (based on their features, scope
and availability) and investigate and review each based upon the following criteria:
•
•
•
•
•
Who/At what level is the spreadsheet auditing assistant designed to help?
How does the spreadsheet auditing assistant work and at what stage can it be
used?
What types of errors is the spreadsheet auditing assistant designed to find?
How successful is the spreadsheet auditing assistant in identifying errors?
How does the spreadsheet auditing assistant compare to the others reviewed?
To perform the investigation and review the following procedures will be used:
•
•
•
Obtain the spreadsheet auditing assistants.
Identify the scope, target users/uses and apparent functionality by using the
software in a ‘free investigation’ session.
Using pre-defined and/or specially created spreadsheets check the spreadsheet
auditing assistant to discover its success at identifying errors.
Secondary Objective
Ø To test the different methods/software tools available against a standard
spreadsheet in an experimental environment with a number of different subjects.
The secondary objective, outlined above, is now unlikely to take place in its original
format. This is because the author is no longer convinced of its benefit to the project
as the emphasis of the project as a whole has moved slightly from focusing on how
successful the spreadsheet auditing assistants are, towards focusing on how the
spreadsheet auditing assistants work in helping to audit spreadsheets. The experiment
will still take place on an individual basis, but the change of focus means the use of
different subjects is now unlikely. The project as a whole has shifted emphasis from
the success of spreadsheet auditing assistants to the way in which spreadsheet auditing
assistants work, as a result of research into the field of spreadsheet errors and
auditing. The attempts made to provide a classification of errors means that the
interesting thing to investigate about spreadsheet auditing assistants would be to
identify which types of errors they are aiming to find and how they go about
identifying these error types. An investigation into the success of these spreadsheet
auditing assistants is a natural progression from the investigation into how they work,
but is now considered beyond the scope of this project.
Work Done
The Project Proposal contained a Gannt Chart of the tasks needed to complete this
project. This chart is reproduced overleaf, and is followed by an update on each of the
tasks scheduled for the first part of the project.
121
Ø Submit the project proposal
This task was completed in full on time.
Ø Attend the spreadsheet lecture
The lecture was attended, and proved to be quite useful, particularly in illustrating the
scope of the Space software. Prior to this meeting the author met Ray Butler and
discussed the project in general, the possibility of obtaining a review copy of Space,
and attending a training course on Space in January. Although this course has
unfortunately been postponed, it is still the authors intention to attend this course as
part of the review of the Space spreadsheet auditing assistant. Ray also offered to
assist in obtaining evaluation copies of the other spreadsheet auditing assistants
needed for review in this project.
Ø
Ø
Ø
Ø
Ø
Investigate spreadsheet history
Investigate spreadsheet error research
Investigate spreadsheet development techniques
Investigate spreadsheet error impact
Investigate spreadsheet auditing research/methods
122
Each of the above tasks were scheduled to run from the start of the project, until
January. These tasks are now virtually complete, with only the option of adding to the
research done available.
Ø
Ø
Ø
Ø
Ø
Write up spreadsheet history
Write up spreadsheet error research
Write up spreadsheet development techniques
Write up spreadsheet error impact
Write up spreadsheet auditing research/methods
Each of the tasks that involve documenting the research done to date in the field of
spreadsheet auditing are due for completion in mid-February 2001. This should be
accomplished by that date.
Ø Write up the Interim Report
Ø Submit the Interim Report
The Interim Report will be completed and submitted by the deadline of 1st February
2001.
Ø Develop sample spreadsheet and scenario for testing
In light of the change in the plan to review the spreadsheet auditing assistants on an
individual basis with the focus being on the methods rather than the results, there have
been wholesale changes to the experiment side of the project. The sample spreadsheet
and scenario will now be based on a similar concept to those used in Panko’s Two
Corpuses of Spreadsheet Errors paper [Panko 2000], The Wall Corpus and The
Galumpke Corpus. There is a possibility that this will change if test spreadsheets
become available from the development of SPACE. The author has contacted Ray
Butler to inquire as to the availability of these spreadsheets. If these are available it
may be preferable to use these as the use of specially designed spreadsheets may
influence the results of the review as the errors are already known. They are also more
likely to provide a real-world slant on the evaluations than would be available through
the purely academic spreadsheets.
Ø
Ø
Ø
Ø
Plan auditing experiments
Do auditing experiments
Review Conclude auditing experiments
Write up reviews/conclusions
Each of the above have been significantly affected by the shift from an experimental,
success based approach to an individual, how it works based approach. As a result the
experiment has been re-scheduled to reflect this. A new project plan has been
formulated and is shown here.
123
Amended Gannt Chart
ID
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
Task Name
Duration
Submit Interim Report
0 days
Write up spreadsheet history
28 days
Write up spreadsheet error research
28 days
Write up spreadsheet error impact
28 days
Write
up
spreadsheet
auditing
28 days
research/methods
Develop/Acquire sample spreadsheet(s)
7 days
for testing
Test Spreadsheet Auditing Assistants
10 days
against sample spreadsheet(s)
Write up test results
7 days
Review/Conclude tests
4 days
Write up conclusions
7 days
Conclude the project
7 days
Academic review
4 days
Write up final report
14 days
Prepare presentation
14 days
Do presentation
0 days
Submit Final Report
0 days
Write paper for EuSpRIG conference
21 days
Present paper at EuSpRIG conference
0 days
124
Start
Thu 01/02/01
Mon 15/01/01
Mon 15/01/01
Mon 15/01/01
Finish
Thu 01/02/01
Wed 21/02/01
Wed 21/02/01
Wed 21/02/01
Mon 15/01/01 Wed 21/02/01
Thu 01/02/01
Fri 09/02/01
Mon 12/02/01
Fri 23/02/01
Mon 26/02/01
Wed 07/03/01
Tue 13/03/01
Thu 22/03/01
Mon 02/04/01
Fri 06/04/01
Tue 13/03/01
Mon 02/04/01
Tue 01/05/01
Tue 01/05/01
Thu 05/07/01
Tue 06/03/01
Mon 12/03/01
Wed 21/03/01
Fri 30/03/01
Thu 05/04/01
Wed 25/04/01
Fri 30/03/01
Mon 02/04/01
Tue 01/05/01
Tue 29/05/01
Thu 05/07/01
Project Direction
For reasons previously discussed, the project has undergone a shift in emphasis,
which has resulted in a change in the project plan from an experimental investigation
into the success of spreadsheet auditing assistants, to an investigation into how
spreadsheet auditing assistants actually work and what they aim to accomplish. This is
not to say that the success of spreadsheet auditing assistants is no longer an issue,
rather that it is logical to identify and analyze exactly how they actually work before
trying to measure their success. The relative success of the different spreadsheet
auditing assistants available is undoubtedly an area for future investigation.
Further Developments
During this project, the author has been introduced to Ray Butler, an employee of the
Inland Revenue who specialises in spreadsheet audits for tax evaluation purpose s, and
is a member of the Programme Committee for the Second International EuSpRIG
(European Spreadsheet Risks Interest Group) Conference on Spreadsheet Risks,
Development, and Audit Methods. As a result of discussions with Ray Butler, in
addition to the authors discussion with the Project Supervisor, Mike O’Hara, a
member of ISACA (Information Systems Audit and Control Association), which
sponsors EuSpRIG, the author, in collaboration with Mike O’Hara, has been asked to
submit a paper for presentation at t he conference in July 2001.
125
Appendix G – Call For Papers for EuSpRIG Conference
Call for Papers
“Controlling the Subversive Spreadsheet”
Second International EuSpRIG Conference on
Spreadsheet Risks, Development and Audit Methods
Vrieje Universiteit, Amsterdam, July 5-6 2001
EuSprig is issuing a Call for Papers for the 2001 conference on Spreadsheet Risks,
Development and Audit Methods.
The theme of the 2001 conference is “Controlling the Subversive Spreadsheet”. The
programme will concentrate on spr eadsheet development and audit tools and methods
We are seeking papers (up to 5000 words) from academics and management
summaries (up to 2000 words) from academics and business people – Spreadsheet
users, developers, auditors and accountants - who can contribute to the prevention,
detection and correction of errors in spreadsheet models and applications.
Timetable
Submission of Abstracts (optional but helpful)
31 December, 2000
Submission of draft papers
31 March 2001
Notification of acceptance
30 April 2001
Submission of finalised papers
20 May 2001
Submission Instructions
Visit www.gre.ac.uk/~cd02/eusprig
or contact David Chadwick, Information
Integrity Research Group, University of Greenwich (D.R.Chadwick@greenwich.ac.uk
)for details of formatting, handling of illustrations etc. Drafts / Abstracts (in English)
should be emailed to David Chadwick in Microsoft Word or Rich Text Format.
Programme Committee
David Chadwick (Chair)
Professor. Margaret van Biene-Hershey
Ray Butler
Luc Kordelwerk
Robert Li
Professor Ray Panko
Barry Pettifor
Harmen Ettema
University of Greenwich
Vrije Universitat, Amsterdam
H M Customs & Excise, Liverpool
Dexia Bank, Belgium
KPMG London
University of Hawaii
PriceWaterhouseCoopers, London
PricewaterhouseCoopers Amsterdam
126
Appendix H – Plagiarism Statement
Form G
BSc Business Information Systems
Final Year Project
Title: Spreadsheet Auditing
Student first name and Initials: David J
Student last name: Nixon
Supervisor Name: Mike O’Hara
Date: 3rd May 2001
PLAGIARISM STATEMENT:
"By signing below, I confirm that I have read and understood the University's
definition of and penalties for plagiarism as contained in the University Regulations
(available at http://www.salford.ac.uk/isi/qmit/dplgrism.htm) and that this report is
my own, original work. Any quotations from the works of other people, whether from
books, articles, the internet or any other sources (including past student projects),
have all been acknowledged by me at the point(s) where I have used that material,
according to the guidelines given on the BIS Project Handbook (available at
http://www.isi.salford.ac.uk/modules/bisproj/".
STUDENT'S SIGNATURE: _________________________________
N.B. This statement, completed and signed by the student, should accompany the
submission of the final report and log book.
127