A Methodology for Evaluating Existing Information Systems As Recordkeeping Systems – 2002 Version The Indiana University Electronic Records Project INTRODUCTION In using this methodology it is important to recognize the collaboration between archivists and the IT community, whose vocabularies at times do not quite match. This methodology is centered around archival and recordkeeping concepts, but facilitated by practices common to the IT community. Perhaps the language is most at odds with the term record. In the IT context a record might be thought of as a data structure representing an element of a file. However, in the context of this methodology, a record is evidence of a business transaction. An understanding of this usage of record is essential for success with the methodology, as is an understanding of other terms (which can be found in the glossary and throughout the text) and of the Functional Requirements and Metadata Specifications (also in the appendix and discussed later in the text). In general, the focus here is on information as it satisfies requirements of authenticity and evidence. Several phases and tasks for the methodology have been identified. Involvement in the information system’s design stage makes the process much easier to implement. In most cases, designing a new system involves incorporating your requirements or specifications and the results of your business process models into the design of the new system. A methodology for analysis of new systems is outlined in a separate document. Analysis of existing systems is normally a more time consuming, more difficult process. It involves not only specifying requirements and metadata specifications and a list of records to be captured. It also requires an analysis of how the present system is managing the data. In essence, the process involves analysis of “what is” as depicted by models and system documentation with “what should be” as defined by the requirements, specifications and models. More specifically, analysis of existing systems includes the following activities: 1) a description and analysis of the business processes by means of a technique known as “modern structured analysis,” 2) a description of how the information system is presently managing records of the identified processes, 3) an evaluation of the information system against the Functional Requirements and Metadata Specifications in the context of the identified business processes, and 4) recommendations for intervention to satisfy the Functional Requirements and Metadata Specifications. Although represented in the text as distinct phases, the reality is that thought processes don't necessarily work in a step-by-step fashion. A methodology may portray a progression through specific steps, but a person using the methodology should be able to consider multiple factors at any point through that progression. The entire methodology document and supporting materials should be reviewed and understood before proceeding with its use. 1 Step 1: DESCRIBING AND MODELING BUSINESS PROCESSES In this initial step of the methodology, the primary goal is to construct a representation of the logical processes of the business, i.e., to create a conceptual model of the work or activities that must be undertaken no matter how the information system is implemented or who does the work. To identify these processes, the methodology uses concepts derived from "modern structured analysis" techniques such as those advocated by Stephen McMenamin, John Palmer, and Edward Yourdan. This form of analysis has been defined as “a process-centered technique that is used to model business requirements for a system. The models are structured pictures that illustrate the processes, inputs, outputs, and files required to respond to business events.” [Jeffrey L. Whitten and Lonnie D. Bentley, Systems Analysis and Design Methods, 4th ed. (Boston: McGraw-Hill, 1998), p. 122. Description of the business requirements is undertaken by means of a process known as decomposition . This process is a means of developing a top-down analysis of the business system, and of breaking down the business system into smaller and smaller processes and sub-processes. In accordance with “modern structured analysis” techniques, the IU methodology decomposes logical processes (business activities that must be undertaken no matter how one implements the system) into three components: Functions, Event Processes or Transactions, and Elementary Processes. In other words, business systems are comprised of functions, which are decomposed into business processes, which ultimately are reduced to business events, which normally represent a single process responding to external and temporal inputs and result in one or more outputs known as elementary processes. In the decomposition process, the activities at the highest end of the business system are known as business functions. A function is a “set of related and on-going activities of the business. A function has no start or end; it just continuously performs its work as needed.” (Whitten and Bentley, Systems Analysis and Design Methods, p. 218). Functions normally are decomposed into sub-functions or into a more discrete and related set of on-going activities. Functions are named with nouns that describe the entire set of activities. Examples of functions and sub-functions from the business area of Office of the Registrar include: Function: Student Recordkeeping - Subfunctions: Official student record maintenance, student degree maintenance, current semester information maintenance. Functions consist of business processes that respond to business events. A business event is “something that ‘happens,’ and that causes business data to change.” Whitten and Bentley, Systems Analysis and Design Methods, 3rd ed., p. 524). An event “is a logical unit of work that must be completed as a whole. An event is triggered by a discrete input and is completed when the process has responded with appropriate outputs. Events are sometimes called transactions.” (Whitten and Bentley, Systems Analysis and Design Methods, p. 218). There are three basic types of inputs that will trigger a business event or transaction: an external event that is initiated by agents outside the system being reviewed, a temporal event that is triggered by the arrival of a predetermined point in time, and a state event that is triggered by a system’s change from one state or condition to another. Most events or transactions are represented by a single 2 process, although occasionally, the event may include two or three processes. An event process or transaction is described in a single sentence that identifies the individual or agency initiating the action (subject-phrase); the official action (verb-phrase); and the individuals or objects acted upon or interacted with (object-phrases). Examples of event processes or transactions taken from the Office of the Registrar work area include: For Subfunction: Student grades and credit maintenance, the event processes or transactions include: 1) Post grades for students upon completion of course work (Input: grade roster from faculty member, and Output: Create a grade and credit record); 2) Assign credit for student work done at other academic institutions (Input: Record of work completed at another institution, and Outputs: Create a credit articulation or evaluation report, and modify student record to reflect the decision). An event process is further decomposed into elementary processes, which are defined as “discrete, detailed activities or tasks required to complete the response to an event.” (Whitten and Bentley, Systems Analysis and Design Methods, 4th ed., p. 219.) In other words, elementary processes are the outputs generated by the business event. Types of elementary processes include: creating a new occurrence of an entity (add), updating an occurrence of an entity (change or modify), and deleting an occurrence of an entity. The methodology omits any processes that do nothing more than move or route data, thus leaving the record unchanged. Elementary processes are named with a strong action verb followed by an object clause that describes the work being performed. Examples of elementary processes from the Financial Aid work area include: Create report listing federal aid recipients with unsatisfactory academic progress, record appeal information in student’s financial aid record, complete work-study verification form received from employer, and provide certification information to the lender. Record creation occurs at the event or transaction level, and the actual records to be analyzed are those documents received as inputs to the system and those records created as a result of the outputs or elementary processes generated in response to the external or temporal event. For example: The business event “processing an appeal” is initiated or triggered by a student or his/her parents, and the input document is the appeal letter received from the student or the parents. The outputs or elementary processes of this event are 1) Making and recording a decision on the appeal, 2) Modifying the student’s financial aid data based on the appeal decision, and 3) Notifying the student about the decision. The appeal letter, the decision document, the modified student record, and the notification are the records of the process. Eventually all this business process information is described or depicted in models or representations that illustrate, usually through the use of pictures or symbols, the various components and relationships of the processes. Models designed to “depict the system independent of any technical implementation” are known as logical models or essential models. (Whitten and Bentley, Systems Analysis and Design Methods, 4th ed., p. 210.) And of the logical models, it is the opinion of the Archives staff that the most valuable models for archivists are those that focus on system processes, specifically business function decomposition diagrams, business event diagrams, and business process data flow models. In the IU methodology, staff normally create functional decomposition and business event diagrams. 3 What types of information are contained in these models, and what do the models look like? To answer these questions, let us review the products Archives personnel created for the business function “Student Recordkeeping.” As a first step, business processes for this function are defined in a short narrative statement. Eventually this information is used to generate a functional decomposition diagram for the function. A partial diagram for the function “Student Recordkeeping”contains the following information and is represented in the following manner. Function: Student Recordkeeping Sub-Function: Semester Data Maintenance Event Process: Dept. Modifies Course Inventory. Event Process: Faculty Modifies Class Schedule Event Process: Student Modifies Course Selections Sub-Function: Student Record Maintenance Sub-Function: Grade and Credit Posting Event Process: Assign Grade for Withdraw al Event Process: Assign Grade for Completion Event Process: Assign Grade for Transfer Work Event Process: Assign Grade Changes Event Process: Assign Grade for Other Credit Sub-Function: Degree Recording Sub-Function: Academic Profile Maintenance Event Process: Modify Academic Profile Sub-Function: Enrollment Certification Event Process: Create Academic Profile Sub-Function: Degree Certification Event Process: Post Dept. List of Degree Recipients Event Process: Post Dept. Certification Data Event Process: Post Certification From IUCare Application 4 Event Process: Post Degrees From IUCare Application Once the functional decomposition diagram is created, staff generate descriptions of the business event processes, including information on the inputs and the various elementary processes or outputs. Initially this data is captured in a simple form that includes the following categories of information for each event process: Name of process, input activities, input record, output activities, and output record(s). Once this data is gathered, staff creates business event diagrams for each of the sub-functions. For the event processes “department modifies course inventory,” and “processing an appeal received from a student,” the models contain the following information and are represented in the following manner. Business Event Diagrams - Sub-function: Semester Information Maintenance - Event Process: Department Modifies Course Inventory Academic Unit Updates to Course Inventory Modification Confirmation Process: Modif y Course Inventory Inventory Updates System Containing Course Inventories 5 Business Event Diagram - Sub-Function Financial Aid Awards - Event Process: Processing an Appeal Received from a Student Notification of Decision Student Process Appeal Appeal Notification Record Appeal Information into Student's Record Modify Student's Record System Containing Student Financial Aid Record System Containing Student's Financial Aid Record Record Decision on Appeal into Student's Record System Containing Student's Financial Aid Record WORK STEPS: 1. Project staff selects a business area for analysis. 2. Analyst reviews existing functional decomposition models, process models or data flow models, event diagrams or event lists, and other available documentation describing business processes. It is particularly important to review process models created when the system was designed. 6 3. Analyst conducts interview(s) with one or more staff from the business area to gather information about major business functions and event processes. Again it is extremely important to keep in mind that what we are asking staff to describe are business requirements and not a list of implementation procedures. Initially, staff to be interviewed should understand the responsibilities of the entire business unit for identification of higher-level functions and events. As needed, other staff may be identified as appropriate resources for identification of elementary processes. Questions to be asked in every case: ** What are the major business functions and subfunctions of this business unit? ** What are the business processes undertaken to implement these functions? In other words, what are the event processes or transactions involved in performing this function? ** What are the business events that trigger an activity and cause records to be produced? ** What are the elementary processes that are initiated in response to these events? These processes will include: creating a new occurrence of an entity (add); updating an occurrence of an entity (change or modify); and deleting an occurrence of an entity. 4. Analyst creates a narrative statements describing 1) the various business processes for the function(s) under review, and 2) each event process transaction, including information on the name of the event process, input activities, and output activities. 5.Analyst creates a functional decomposition diagram that depicts the relationships between and among functions and business events or transactions for the function(s) under review. 6. Analyst creates models or depictions of the business event processes, including information on the inputs and the various elementary processes or outputs. 7. Analyst creates a list of the records that are created as products of the processes under review. 8. Analyst compares any logical models of business processes created when the system was designed with the business models generated during interviews with system managers and identifies and describes any differences in the two models. 9. If there are differences in the definition of processes and records creation, the analyst will work with record creators and data managers to reconcile difference and come to a agreement on the products of the business processes. WORK STEPS: TIPS Examine functions proposed at a particular level to see if they fit within a higher level function. Even a major business area typically has only six to twelve firstlevel functions. Second-level functions typically have between three and eight third-level functions. (Low numbers are very common.) Be as comprehensive and complete as possible. Assure that the list adequately accounts for all major activities of the area. An outline form is appropriate for 7 representing the relationships between functions and sub-functions. As with an outline, balance is expected but not symmetry. Functions at the same level should have roughly the same significance, complexity, etc. However, one second-level function may have two or three third-level functions while others may have eight or nine. Without experience it is difficult to tell when the functional decomposition is complete. The function list should be reasonably complete, but will not be exhaustive. In general a third-level decomposition should be sufficient, but it may be necessary to go further to gain enough detail for the identification of transactions in the next task. REQUIRED MATERIALS: 1. Functional Decomposition models, Process models, or other descriptions of the business requirements. 2. Other documentation that depicts or models the business requirements, such as Event lists, Event-Response models, and Data Flow models. DELIVERABLES: 1. Decomposition descriptions and diagrams of business functions, event processes or transactions, and elementary processes currently being undertaken. 2. Lists of the records that are being produced by the elementary processes. 3. If required, descriptions or depictions of how the current business processes differ from the set of business requirements created in the systems design stage. 4. If required, a description of how differences in the analysis of business processes were resolved. ROLES AND RESPONSIBILITIES: Conducted by: project analyst Reviewed by: project staff, business area staff Approved by: project staff Information resources: business area staff 8 Step 2: ANALYZING THE SYSTEM IN TERMS OF THE IU RECORDKEEPING REQUIREMENTS The goals of this step in the methodology are twofold: a) describe or model how the existing system is presently managing the records created by the business processes identified during the analysis conducted in Step 1; and b) analyze these results in terms of the IU Functional Requirements. In defining how the system is presently managing records, the analyst will rely heavily on relevant documentation on system functionality and operations. Where documentation is non-existent or lacks detail, the analyst will interview knowledgeable staff who understand how data is processed and managed in the system. The goal in this step is to determine how the system is how the system is managing the records under review. To determine this, the analyst will ask a series of questions derived from the IU Functional Requirements document. The Functional Requirements are system level requirements, and therefore are meant to be applied at a much higher level than the individual record. In other words, for all the Functional Requirements the analyst will begin by reviewing and analyzing how the requirement is met at the highestlevel sub-function for the business function under review. For example, in the decomposition model depicted above, this would mean analyzing as a body all records produced by the three sub-functions and the six business transactions for the high-level sub-function “Degree Recording.” If during the analysis it becomes clear that the records produced in the course of completing lower-level sub-functions are managed differently than the records of related sub-functions, the analyst will then proceed to analyze the records at the next lower level. For example, in reviewing the system for the requirement “Authenticity,” the analyst discovers that rules for modification of records for the subfunction “Enrollment Certification” are different than for the sub-function “Degree Certification.” Once this difference is discovered, the analyst would immediately adopt a strategy of reviewing separately the products of business processes for each of the lowerlevel sub-functions. Similarly, if different procedures are undertaken at the level of the business transaction, then the analyst will begin the analysis of the system for that requirement at the level of the business event or record level. However, this will be a rare occurrence. In the vast majority of cases, the analysis of the system in terms of the IU Functional Requirements will be at the highest sub-function level. WORK STEPS: 1. Analyst gathers available documentation on systems, standards, procedures, retention schedules, etc. Prominent categories of documentation include: Processing descriptions with models, if available; procedure manuals and workflow models relating to routing, inputting, updating, saving and deleting records; procedure manuals relating to backingup, migrating, purging, exporting and restoring data; documentation on data and data models to determine what types of informational value may be present in records; procedures that define access and use of records, and training procedures; existing 9 disposition schedules and laws, policies and best practices related to recordkeeping; policies and procedures dealing with security and authorization mechanisms; documentation describing predefined reports and inquiries; and documentation describing specific applications that are part of the system, including on-line processing transactions and batch jobs. 2. Where documentation is unavailable or lacks details, the analyst interviews staff and administrators who are familiar with the how the system processes and manages data and records. 3. Using the functional decomposition analyses and system documentation, the analyst reviews how the system is managing records in accordance with the “Requirements for Electronic Records Management Systems.” REQUIRED MATERIALS: 1. Functional decomposition analyses from step 1. 2. System documentation. 3. Other documentation (e.g., procedure manuals, policies, retention schedules). 4. Notes from interview with staff and administrators 5. IU Functional Requirements statement. DELIVERABLE: 1. This document will be organized at the highest level sub-function, and only will include analysis at lower levels as needed. Within each sub-function, the responses will be arranged according to the list of Functional Requirements and will address the issues defined for each requirement. For each requirement, prepare a brief narrative statement describing how the system does or does not meet the requirement. ROLES AND RESPONSIBILITIES: Conducted by: project analyst Reviewed by: project staff, business area staff, computing services staff Approved by: project staff Information Resources: business area staff, computing services staff 10 STEP 3: ANALYZING HOW THE SYSTEM IS DOCUMENTING RECORDS IN TERMS OF THE IU METADATA SPECIFICATIONS The goals of this step in the methodology are twofold: a) describe or model how the existing system is presently documenting the records created by the business processes identified during the analysis conducted in Step 1; and b) analyze these results in terms of the IU Metadata Specifications. In essence the goal is to determine if the “evidence” required to document the business transactions exists and in what form. In other words, the primary objective is to determine whether the metadata category or element exists for that record or class of records. The goal is not to determine whether the value provided for that metadata element is correct or incorrect. As in the previous step, the analyst will rely heavily on relevant documentation to determine how the system is presently documenting records. Where documentation is non-existent or lacks detail, the analyst will interview knowledgeable staff who understand how data is processed and managed in the system. In reviewing the documentation and determining how the system is documenting records, the analyst will be guided by the specifications for recordkeeping listed in the Metadata Specifications statement. Records within business events and even business subfunctions often will include the same types of metadata. This is particularly true for socalled “management” metadata that document why and how records will be accessed and used, disposed of, and preserved. In most cases, the type of metadata collected to document these activities will be the same for many, many records within a business process. Even audit trail metadata documenting activities performed on individual records is predictable because so much of this type documentation is collected automatically by the system and applied to many records within a business process. Finally even types of metadata that are unique to a specific record, such as the unique identifier, can be analyzed at the aggregate level by asking the question: for records of this class or function, does the system assign a unique identifier. Again, it is important to remember that what we are analyzing is whether the system collects or creates this category of metadata and not whether the metadata value is correct or not. Accordingly, as with the functional requirements, the analyst will begin by reviewing and analyzing how the metadata specification is met at the highest-level sub-function for the business function under review. If during the analysis it becomes clear that the records produced in the course of completing lower-level sub-functions are documented differently than the records of related sub-functions, the analyst will then proceed to analyze the records at the next lower level. For example, in reviewing the system for “Disposition” Metadata,” the analyst discovers that within the sub-function “Enrollment Certification” retention periods are specified, while for the related sub-function “Degree Certification” retention metadata is not present. Once this difference is discovered, the analyst would immediately adopt a strategy of reviewing separately the products of business processes for each of the lower-level sub-functions. Similarly, if types of metadata collected at the level of the business transaction are different, then the analyst will begin the analysis of the system for that specification at the level of the business event or record level. 11 It is also important to determine the nature of the logical relationship between the record content and the metadata. Is the metadata part of the record? Is it electronically linked to the record? Is the metadata a paper record whose location is electronically linked to the record? Or is there no logical relationship between the metadata and record? WORK STEPS: 1. Analyst gathers available documentation on how the system documents data and records. Prominent types of documentation include business process models, data models, and data dictionaries. 2. Where documentation is unavailable or lacks details, the analyst interviews staff and administrators who are familiar with the how the system documents data and records. 3. Analyst reviews how the system is documenting records in accordance with the IU Recordkeeping Metadata Specifications. REQUIRED MATERIALS: 1. Functional decomposition analyses from step 1. 2. Documentation on how the system is documenting records. 3. Notes from interview with staff and administrators 4. IU Metadata Specifications document. DELIVERABLE: Responses will be organized at the highest level sub-function, and only will include analysis at lower levels as needed. Within each sub-function, the responses will be arranged according to the list of Metadata Specifications and will address the following questions: a) Does the metadata exist? B) What is the logical relationship between the metadata or a citation to the metadata and the record content? ROLES AND RESPONSIBILITIES: Conducted by: project analyst Reviewed by: project staff, business area staff, computing services staff Approved by: project staff Information Resources: business area staff, computing services staff 12 Step 4: EVALUATE THE SYSTEMS IN TERMS OF THE "FUNCTIONAL REQUIREMENTS FOR EVIDENCE IN RECORDKEEPING" How effectively does the existing system satisfy the requirements of a recordkeeping system? In this step, the results of the analyses conducted in steps 1 and 2 are evaluated in terms of various categories of compliance. WORK STEPS: 1. Project staff review documentation produced by analyst in previous steps. 2. Project staff interview analyst, if necessary. 3. It may be necessary to conduct additional interviews with record creators or to review the documentation so as to gather more specific or detailed information. Even with the best efforts of the analyst in preparing documentation for the evaluation, the evaluator(s) may require additional information or interpretation in order to evaluate correctly the evidence collected during the previous analyses. 4. For each sub-function the project staff evaluates the systems according to the following criteria: How effectively does the system satisfy the requirements of a recordkeeping system? For each of the Indiana University Functional Requirements, what evidence is there that the system satisfies that Requirement? Do not respond with a simple yes or no; generate a brief narrative statement with specific examples of evidence that the requirement is fulfilled or not fulfilled. In addition, classify the level of compliance in terms of one of the following three categories: 1) satisfied, 2) partially satisfied, and 3) not satisfied. REQUIRED MATERIALS: 1. Documentation from steps 1 and 2. 2. Other supporting documentation accumulated in previous phases. 3. Indiana University versions of Functional Requirements. DELIVERABLE: 1. A document that for each high-level sub-function that describes the nature and level of compliance with each of the IU Functional Requirements. ROLES AND RESPONSIBILITIES: Conducted by: project staff Reviewed by: archivist Approved by: archivist Information resources: project analyst 13 Step 5: EVALUATE THE SYSTEMS IN TERMS OF HOW WELL THEY MEET THE IU METADATA SPECIFICATIONS How effectively does the existing system satisfy the IU Metadata Specifications. In this step, the results of the analyses conducted in steps 1 and 2 are evaluated in terms of various categories of compliance. WORK STEPS: 1. Project staff review documentation produced by analyst in previous steps. 2. Project staff interview analyst, if necessary. 3. It may be necessary to conduct additional interviews with record creators or to review the documentation so as to gather more specific or detailed information. Even with the best efforts of the analyst in preparing documentation for the evaluation, the evaluator(s) may require additional information or interpretation in order to evaluate correctly the evidence collected during the previous analyses. 4. For each business event the project staff evaluates the systems according to the following criteria: Is the appropriate metadata (context, structure, and content) being captured and preserved inviolate by the existing system? For each of the Indiana University Metadata Specifications, what evidence is there that the system satisfies the Specification? Again, do not respond with a simple yes or no. Create a brief narrative statement with specific examples of compliance or non-compliance. In addition, classify the level of compliance in terms of one of the following categories: a) Metadata is available electronically with use of record; b) Metadata is available in electronic or paper format, and there is a logical relationship between the metadata or a citation to this metadata and the record itself; c) Metadata is available in electronic or paper format, but there is no logical relationship between the metadata or a citation to the metadata and the record itself; d) Metadata is not available anywhere. REQUIRED MATERIALS: 1. Documentation from steps 1 and 2. 2. Other supporting documentation accumulated in previous phases. 3. Indiana University versions of the Metadata Specifications DELIVERABLE: 1. A document that for each event process or transaction describes the nature and level of compliance with the IU Metadata Specifications. 14 ROLES AND RESPONSIBILITIES: Conducted by: project staff Reviewed by: archivist Approved by: archivist Information resources: project analyst Step 6: RECOMMENDATIONS Once the analysis of how the information system is managing records of the various business processes is completed, staff meet to determine what recommendations will be made for improving the performance of the system. We recognize that not all the problems identified in the evaluation process will be of equal rank or of the same level or degree of seriousness. Therefore there are three categories or levels of recommendations: Highest Priority Recommendations, Concerns, and For Your Information. When making recommendations for changes to recordkeeping systems keep in mind the costs/risks and benefits associated with implementing that particular recommendation. We also recognize that institutions will not likely implement every recommendation we make. The decision to implement will be based on a variety of factors, including an appraisal of the value of the records, costs and benefits, risk of retaining or disposing of documentation, and organizational needs and priorities. WORK STEPS: 1. Project staff reviews the evaluation document for evidence of functional requirements and metadata specifications that have not been met. 2. Project staff prepares a set of recommendations for improving the system based on the three categories listed above. 3. Project staff prepares a report and presents the recommendations to appropriate parties. REQUIRED MATERIALS: 1. The evaluation document from Step 3. 2. Documents from other phases if necessary 15 DELIVERABLES: Report of recommendations in the following format: Introductory statement describing the scope and objectives of the evaluation process Specific recommendations organized into the categories of: Highest Priority Recommendations, Concerns, and For Your Information. Each recommendation will include an explanation of the problem and recommendations on how the unit might address it. This document along with a short cover letter outlining the process is then forwarded to the appropriate data stewards and managers for their review. Soon after transmittal of the report, a meeting will be scheduled to discuss the various recommendations and implementation strategies. ROLES AND RESPONSIBILITIES: Conducted by: project staff, project analyst Reviewed by: archivist, business area staff Approved by: archivist Accepted or Rejected by: business area staff Information resources: project analyst 16