section i.3c – use of computer assisted audit tools
advertisement
CGIAR Centers Internal Audit Audit Manual – Section I.3C SECTION I.3C – USE OF COMPUTER ASSISTED AUDIT TOOLS AND TECHNIQUES Introduction The CGIAR Centers are heavily dependent on automated information systems to capture, store and transmit data, to support work processes, to provide operational and financial reports to aid management monitoring and decision making, and to facilitate internal and external communications. It is therefore important that the overall internal audit coverage for a Center include assurance and advisory work related to these areas. Computer Assisted Audit Tools and Techniques (CAATTs) may be used to supplement manual techniques as part of audits which have as an objective the evaluation of the confidentiality, integrity, availability and reliability of electronic data and the systems that process, store and report such data. CAATTs are, in the broadest sense of the term, any use of a computer during the audit. In practice, however, the term CAATTs has become synonymous with specialized use of tools which (a) incorporate data analytics into the audit process; (b) automate sample selections from large electronic datasets, or (c) test system vulnerabilities in terms of availability and integrity of the information accessed by the system, or data security/privacy. Sometimes audit support tools such as audit management systems and electronic working paper systems are included under the heading of CAATTs. However they are best distinguished as Beneficial Electronic Audit Support Tools (BEASTs) and are not dealt with in this Manual Section. CAATTs may produce a large proportion of the audit evidence developed on audits and, as a result, the auditor should carefully plan for and exhibit due professional care in the use of CAATTs. A CAATTs‐driven review is limited only to the data saved on files in accordance with a systematic pattern. Much data is never documented this way. In addition saved data often contains deficiencies, is poorly classified, is not easy to get, and it might be hard to become convinced about its integrity. So CAATTs are a complement to an auditorʹs tools and techniques, not the only one. In certain audits CAATTs canʹt be used at all. But there are also audits which simply canʹt be made with due care and efficiently without the use of CAATTs. Version: August 15, 2009 Page I.3C: 1 CGIAR Centers Internal Audit Audit Manual – Section I.3C Information systems are not uniform throughout the CGIAR Centers. They vary in the degree of automation of processes and in the type of systems acquired or developed. Thus the applicability of the potential CAATTs which can be used, and the protocols for their implementation, must be assessed on a Center‐by‐ Center basis. Most Frequent Uses of CAATTs analytical review of total populations of data in information and communication technology (ICT) systems testing of balances and underlying transactions in ICT systems compliance tests of ICT system general controls compliance tests of ICT system application controls recovering and decrypting data penetration testing of network, operating and application system security decision making using audit expert systems integrity checking of data held on databases Types of CAATTs CAATTs include many types of tools and techniques, some of which are more relevant to the CGIAR Center environment than others. The main types are: Type Generalized Audit Software: Comments The CGIAR Internal Auditing Unit, and some Center Internal Auditing Units, have acquired ACL as their primary generalized audit software. The advantage of this software is that it can be used across a wide variety of systems and therefore Centers. A computer program or series of programs designed to perform certain automated functions. These functions include reading computer files, selecting data, manipulating data, sorting data, summarizing data, performing calculations, selecting samples, and printing reports or letters in a format specified by the auditor. This technique includes use of: In addition, some Centers financial / Commercially available generalized data enterprise resource management extraction and analysis software (the two systems have auditing features main such products are currently ACL which can be utilized. and IDEA), Version: August 15, 2009 Page I.3C: 2 CGIAR Centers Internal Audit Audit Manual – Section I.3C SQL query language, Crystal reporting Excel spreadsheet and Access applications Audit software embedded in production systems. In house developed audit software The report writer that comes with the application Audit specialized software can easily perform the following functions across a whole data population: Data queries Data stratification. Sample extractions. Missing sequence identification. Statistical analysis. Benford’s Law analysis Recalculations. Duplicate inquires. Pivot tables. These type of CAATTs make remote or continuous auditing possible. However given the nature and fragmentation of systems across the CGIAR Centers investment to implement continuous auditing approaches will probably not pass cost‐benefit tests. At present, generalized audit software is best used on an ad hoc basis for particular audit assignments. Utility Software: Computer programs provided by a computer hardware manufacturer or software vendor and used in running the system. Utility software can be employed as a CAATT to examine processing activity, test programs, system activities and operational procedures, evaluate data file activity, and analyze job accounting data. Version: August 15, 2009 When using utility software, the auditor should confirm that no unplanned interventions have taken place during processing and that the utility software has been obtained from the appropriate system library. The auditor should also take appropriate steps to protect the integrity of the Center’s system and files since these utilities can easily damage the system and its files. The auditor would normally only has ‘read only’ access. Page I.3C: 3 CGIAR Centers Internal Audit Audit Manual – Section I.3C Application Software Tracing and Mapping: The auditor should be aware that application software tracing and Specialized tools that can be used to analyze mapping only points out the the flow of data through the processing logic of potential for erroneous processing; it the application software and document the does not evaluate actual production logic, paths, control conditions, and processing data. sequences. Both the command language or job control statements and programming language When using application software can be analyzed. This technique includes tracing and mapping, the auditor program/system: mapping, tracing, snapshots, should confirm that the source code parallel simulations, and code comparisons. being evaluated generated the object program currently being used in production. Test Data: When using test data, the auditor should be aware that test data only Simulated transactions that can be used to test point out the potential for erroneous processing logic, computations and controls processing; this technique does not actually programmed in computer applications. evaluate actual production data. The Individual programs or an entire system can be auditor also should be aware that tested. This technique includes Integrated Test test data analysis can be extremely Facilities (ITFs) and Base Case System complex and time consuming, Evaluations (BCSEs). depending on the number of transactions processed, the number of programs tested, and the complexity of the programs/system. Before using test data the auditor should verify that the test data will not permanently affect the live system. Data recovery and decryption tools: Various tools can be used to retrieve deleted data, data from damaged computer hard drives, and to decrypt protected data. Version: August 15, 2009 These CAATTs are primarily used for investigations where forensic audit techniques are required. Beyond the simple step of reviewing recycle boxes to find “deleted” data, or the use of the simpler desktop tools to retrieve deleted information, access to specialist tools and expertise for data recovery and decryption will be obtained by CGIAR Internal Auditors on an outsourced basis for specific Page I.3C: 4 CGIAR Centers Internal Audit Audit Manual – Section I.3C assignments. The requirements for such are not sufficiently large to invest in house either at Center or CGIAR IAU level in developing the capacity in the use of these CAATTs and purchasing specialized tools for this purpose. System hacking tools: Applications used in penetration testing to identify and analyze network or application system security vulnerabilities. As indicated later in this Manual Section, the CGIAR internal auditors do not conduct Center system penetration testing using hacking tools. However they encourage and review the results of Center‐ commissioned penetration testing using third party experts. General Considerations in the Use of CAATTs Ref. Policy and Practice Requirements I.3C‐1 Policy: When planning audits, the Auditor in Charge shall consider whether manual techniques should be supplemented with CAATTs. IIA Standards and other references Standard: 1220.A2 In exercising due professional care the internal auditor should consider the use of technology‐based audit and Discussion: other data analysis In determining whether to use CAATTs, the techniques. factors to be considered include: o Computer knowledge, expertise, and ISACA Audit and experience of the auditor Assurance Guideline G3. o Availability of suitable CAATTs and ICT facilities o Efficiency and effectiveness of using CAATTs over manual techniques o Time constraints o Integrity of the information system and Version: August 15, 2009 Page I.3C: 5 CGIAR Centers Internal Audit Audit Manual – Section I.3C ICT environment o I.3C‐1:1 Level of audit risk The CGIAR IAU and some Center IAUs have acquired ACL software, to provide an option for using generalized audit software tool. Practice Requirement: In considering whether to start using CAATTs to analyze and test data in particular systems, consideration shall be given to planned changes to the system that might affect the future usefulness of the CAATTs. ISACA Audit and Assurance Guideline G3 Discussion: The development of CAATTs for particular systems involves an investment of time. The cost‐benefit of this may be affected by planned system changes. The impact of planned changes should be assessed before embarking on this investment. I.3C‐1:2 Practice Requirement: When acquisition of new financial and other corporate systems which interface with financial systems are being planned, the Head of Internal Audit should explore with the Center the potential for acquisition of audit modules available for that system. I.3C‐1:3 Practice Requirement: The use of CAATTs other than generalized audit software (such as ACL or IDEA), such as utility software, customized queries or scripts, and embedded software added to the system for “continuous auditing” activities, shall only be approved by the Head of Internal Audit after consultation with Center IT management and the system owners where this resides outside of the IT group. Version: August 15, 2009 Page I.3C: 6 CGIAR Centers Internal Audit Audit Manual – Section I.3C I.3C‐2 Policy: When a decision has been made to use ISACA Audit and CAATTs, a detailed plan shall be prepared, as Assurance Guideline G3. part of the preliminary survey phase of the audit, to ensure the CAATTs are used effectively and efficiently I.3C‐2:1 Practice Requirement: The major steps to be undertaken by the auditor in preparing for the application of the selected CAATTs shall comprise the following: Set the objectives for using the proposed CAATTs in relation to the overall audit objectives Determine the accessibility and availability of the Center’s ICT facilities, programs/system and data and make appropriate arrangements with the auditee for this Clearly understand the composition of data to be processed, including the quantity, type, format and layout Understand the relationship between data tables where a database is to be examined Define the procedures to be undertaken (e.g., statistical sampling, recalculation, confirmation) Define output requirements Determine resource requirements, i.e., personnel, ICT support Define protocols for obtain access to the organizations ICT facilities, programs/system, and data, including file definitions Document CAATTs to be used, including objectives, high‐level flowcharts, and run instructions ISACA Audit and Assurance Guideline G3. Version: August 15, 2009 Page I.3C: 7 CGIAR Centers Internal Audit Audit Manual – Section I.3C Discussion: I.3C‐3 The Auditor should ascertain the period for which Center data files, such as detailed transaction files, are retained. The auditor should make arrangements with the auditee for the retention of the data to be anaylsed using CAATTs covering the appropriate audit time frame. Access to the Center’s ICT facilities, programs/system, and data, should be arranged for well in advance of the needed time period in order to minimize the effect on the production environment. The auditee should understand the purpose, scope, timing and goals of the CAATTs. Setting clear expectations at the outset is important and these should be communicated during the planning phase. The auditor should assess the effect that changes to the production programs/system may have on the use of the CAATTs. In doing so, the auditor should consider the effect of these changes on the integrity and usefulness of the CAATTs, as well as the integrity of the programs/system and data used by the Auditor. Policy: The Auditor shall obtain reasonable ISACA Audit and assurance of the integrity, reliability, Assurance Guideline G3 usefulness, and security of the CAATTs through appropriate planning, design, testing, processing and review of documentation. This shall be done before reliance is placed upon the CAATTs. Discussion: The use of commercial audit software packages like ACL or IDEA mitigate software integrity, reliability and security risks. These packages contain many controls to assure these factors, and are Version: August 15, 2009 Page I.3C: 8 CGIAR Centers Internal Audit Audit Manual – Section I.3C subject to extensive quality assurance review and user testing. I.3C‐3:1 The availability of many established testing routines for these software packages also mitigate the risks of appropriateness for use. Internal Auditors are encouraged to make use of these routines to assure effectiveness and efficiency in the use of the audit software. Practice Requirement: Where CAATTs are used to extract information for data analysis the auditor shall, as part of the planning process, verify the integrity of the information system and ICT environment from which the data are extracted. ISACA Audit and Assurance Guideline G3 Discussion: I.3C‐3:2 The protocols for data downloads from Center systems, for analysis by audit software such as ACL or IDEA, should be agreed with Centers. The protocols should address controls over the completeness and integrity of the data being downloaded, so that the correct population is obtained. Practice Requirement: CAATTs can be used to extract sensitive program/system information and production data that should be kept confidential. The auditor shall safeguard the program/system information and production data with an appropriate level of confidentiality and security. In doing so, the Auditor shall consider the level of confidentiality and security required by the organization owning the data and any relevant legislation (such as privacy laws). I.3C‐3:3 Practice Requirement: The auditor shall use and document the results of appropriate procedures to provide for the Version: August 15, 2009 ISACA Audit and Assurance Guideline G3 ISACA Audit and Assurance Guideline G3 Page I.3C: 9 CGIAR Centers Internal Audit Audit Manual – Section I.3C ongoing integrity, reliability, usefulness, and security of the CAATTs. Discussion: For example, this should include a review of program maintenance and program change controls over embedded audit software to determine that only authorized changes were made to the CAATTs. When the CAATTs reside in an environment not under the control of the auditor, an appropriate level of control should be in effect to identify changes to the CAATTs. When the CAATTs are changed, the auditor should obtain assurance of their integrity, reliability, usefulness, and security through appropriate planning, design, testing, processing and review of documentation before reliance is placed on the CAATTs. I.3C‐3:4 Practice Requirement: The use of CAATTs shall be controlled by the auditor to provide reasonable assurance that the audit objectives and the detailed specifications of the CAATTs have been met. ISACA Audit and Assurance Guideline G3 Discussion: The auditor should: Perform a reconciliation of control totals if appropriate Review output for reasonableness Perform a review of the logic, parameters or other characteristics of the CAATTs Review the Center’s general ICT controls which may contribute to the integrity of the CAATTs (e.g., program change controls and access to system, program, and/or data files) Version: August 15, 2009 Page I.3C: 10 CGIAR Centers Internal Audit Audit Manual – Section I.3C I.3C‐3:5 Practice Requirement: When using generalized audit software to access the production data, the auditor shall take appropriate steps to protect the integrity of the organization’s data. ISACA Audit and Assurance Guideline G3 Discussion: At this time, there are no plans to develop customized CAATTs. Using generalized software such as ACL or IDEA on downloaded production data for analysis and testing is a safe method. The downloading from Center systems will need to be made by auditee IT staff, under suitably controlled conditions, to ensure that this process does not also affect the production environment. I.3C‐4 Policy: Based on the detailed CAATTs plan prepared during the preliminary survey phase, the use of CAATTs should be identified in the audit terms of reference. I.3C‐5 Policy: The step‐by‐step CAATTs process should be sufficiently documented in the audit working papers to provide adequate audit evidence. ISACA Audit and Assurance Guideline G3. Discussion: Planning Documentation should include: o CAATTs objectives o CAATTs to be used o Controls to be exercised o Staffing and timing Execution Documentation should include: o CAATTs preparation and testing Version: August 15, 2009 Page I.3C: 11 CGIAR Centers Internal Audit Audit Manual – Section I.3C procedures and controls I.3C‐6 o Details of the tests performed by the CAATTs o Details of inputs (e.g., data used, file layouts), processing (e.g., CAATTs high‐level flowcharts, logic) and outputs (e.g., log files, reports) o Listing of relevant parameters or source code Audit Evidence Documentation should include: o Output produced o Description of the audit analysis work performed on the output o Audit findings o Audit conclusions o Audit recommendations Generalized audit software and utility software will automate some or all of the documentation of this process in a controlled manner. In such cases the reports and audit trails produced by these softwares can provide some or all of the required documentation Policy: The audit report should contain a clear description of the CAATTs used. ISACA Audit and Assurance Guideline G3. Discussion: This description should not be overly detailed, but it should provide a good overview for the reader. Depending on the extent to which the CAATTs are used, the description may be presented in the objectives, scope and methodology section of the report, the sections of the report dealing with the Version: August 15, 2009 Page I.3C: 12 CGIAR Centers Internal Audit Audit Manual – Section I.3C subject matter for which CAATTs were used, or in an annex to the report with a more detailed description. Use of CAATTs for penetration testing Penetration testing is a method of evaluating the security of information systems/networks that have external (internet facing) deployment, by simulating an attack by a malicious user (often referred to as “black hat hackers” or “crackers”). The intent of a penetration test is to determine feasibility of an attack and the amount of business impact of a successful exploit, if discovered. The process involves an active analysis of the system for any potential vulnerabilities that may result from poor or improper system configuration, known and/or unknown hardware or software flaws, or operational weaknesses in process or technical countermeasures. This analysis is carried out from the position of a potential attacker, and can involve active exploitation of security vulnerabilities. Automated (computer‐assisted) tools are part of the toolkit for remote penetration testing via the internet and may be used for: • • • • • • • • • • • • • • Network surveying Port scanning Services identification System fingerprinting Vulnerability research and verification Internet application testing Router testing Trusted systems testing Firewall testing Intrusion detection system testing Containment measures testing Password cracking Denial of service testing Modem testing Penetration testing may also include tests for vulnerabilities via non‐internet means, such as from tapping into wireless systems (such as 802.11 wireless, Bluetooth, and infrared systems) or capturing and recreating data from electromagnetic radiation emanating from system devices. Security risks from these sources are most prominent for organizations dealing with highly classified Version: August 15, 2009 Page I.3C: 13 CGIAR Centers Internal Audit Audit Manual – Section I.3C information, significant commercial secrets, or the provision of major physical or financial infrastructure to the local or international economies. These organizations would be of particular interest to spies, terrorists and thieves having the necessary resources to attempt penetrations through such methods. Penetration testing in this area can be expensive and risky, and the vulnerability of attack from such sources is considered low for the CGIAR Centers, and therefore not promoted by Internal Audit at this time. Web‐based penetration tests can be conducted in several ways. The most common difference is the amount of knowledge of the implementation details of the system being tested that are available to the testers. Black box testing assumes no prior knowledge of the infrastructure to be tested. The testers must first determine the location and extent of the systems before commencing their analysis. At the other end of the spectrum, white box testing provides the testers with complete knowledge of the infrastructure to be tested, often including network diagrams, source code and IP addressing information. There are also several variations in between, often known as gray box tests. Penetration tests may also be described as ʺfull disclosureʺ, ʺpartial disclosureʺ or ʺblindʺ tests based on the amount of information provided to the testing party. The relative merits of these approaches are debated. Black box testing simulates an attack from someone who is unfamiliar with the system. White box testing simulates what might happen during an ʺinside jobʺ or after a ʺleakʺ of sensitive information, where the attacker has access to source code, network layouts, and possibly even some passwords. Penetration tests may be conducted as “blue teaming” i.e. with the knowledge and consent of the organization’s IT staff, or “red teaming” i.e. with only the knowledge and permission of upper management. Red teaming is more expensive and complex to manage, but can provide a better indication of day to day security as the system administrators will not be on heightened awareness. The Open Source Security Testing Methodology Manual (OSSTMM), which is managed by Institute for Security and Open Methodologies (ISECOM), a US and Spanish registered non‐profit body, provides an open source peer‐reviewed methodology for performing security tests and metrics, taking into account various national and international laws and sources of standards and guidelines, including those of the US National Institute of Standards and Technology (NIST). Version: August 15, 2009 Page I.3C: 14 CGIAR Centers Internal Audit Audit Manual – Section I.3C The Manual can be downloaded from http://www.isecom.org/osstmm/ Penetration testing is one aspect of a broad security testing methodology promoted in the OSSTMM test cases. The OSSTMM focuses on the technical details of exactly which items need to be tested, what to do before, during, and after a security test, and how to measure the results. OSSTMM is also known for its Rules of Engagement which define for both the tester and the client how the test needs to properly run. An alternative framework known as the Information Systems Security Assessment Framework (ISSAF), which is also intended to address standards for penetration testing, is under development by the Open Information Systems Security Group (OISSG), a US‐based non‐profit organization. A draft framework can be downloaded from http://www.oissg.org/issaf/index.php Ref. Policy and Practice Requirements IIA Standards and other references I.3C‐7 Policy: Penetration testing of Center information systems will not normally be carried out by Internal Audit. Rather, Centers will be encouraged to commission suitably qualified third party specialists to carry out such testing as part of their business continuity (BCP) and information system resilience programs. Discussion: The internal audit groups within the CGIAR do not have the in house capacity to conduct these specialized and potentially high risk audit procedures. They are also more appropriately commissioned directly by Centers as part of their BCP/information system resilience programs. Third party specialists should be selected on the basis of criteria developed in accordance with external benchmarks such as those set out in the OSSTMM. Version: August 15, 2009 Page I.3C: 15 CGIAR Centers Internal Audit Audit Manual – Section I.3C I.3C‐7:1 Practice Advisory: Where Centers have commissioned penetration testing, Internal Audit shall review the testing, results and follow up actions as part of its evaluation of BCP/information system resilience programs, and as input into the assessment of Center enterprise risk management systems. Discussion: I.3C‐7:2 An effective program of ongoing penetration testing by an expert third party, suitably tailored to the type of information security risks faced by the Centers, will provide assurance over the effectiveness of controls over system security. Internal Audit may, where such penetration testing has not been done, or is considered insufficient or out of date, make recommendations to the Centers to implement penetration testing using an expert third party. Penetration testing is also being done or being planned for the Centers under the CGIAR Enterprise Security and Business Continuity Project. Practice requirement: Where Centers have or will be commissioning penetration testing, internal audit should review the arrangements and methodology against agreed benchmarks such as the OSSTMM, to ensure that such testing is carried out effectively and that the risks of the testing are appropriately managed. Version: August 15, 2009 Page I.3C: 16
