IMPLEMENTING AN ELECTRONIC RECORDS MANAGEMENT PROGRAM Philip C. Bantin Indiana University Archivist bantin@indiana.edu IU Electronic Records Program Website: http://www.indiana.edu/~libarch /ER/NHPRC-2/index.html OUTLINE OF PRESENTATION 1) Requirements for an Electronic Records Management System (ERMS) 2) Why On-Line Transaction Processing (OLTP) Systems, Data Warehouses, and Electronic Document Management Systems (EDMS) Do NOT Meet These Requirements OUTLINE OF PRESENTATION 3) Capturing Records – Strategies ** Business Process Engine – Example: European Registry System ** Workflow or Routing Engine – Example: IU Workflow Engine ** Records Management Applications (RMA) – Example: TRIM 4) Capturing Records: Metadata Specifications Information Systems Systems Development Lifecycle System concept: purpose, goals, scope Analysis: user/functional requirements Design data design: what information? software design: processed how? interface design: user interaction? Coding and testing: execute & evaluate Key issue: Systems do (only) what they’re designed to – purpose, goals, scope, requirements. Data and Information Systems Transaction Processing Decision Support Data-oriented Automate basic business processes MIS/EIS Data warehouse Electronic Document Management Systems (EDMS) Transaction Processing Systems (TPS) Employing DBMS Software The most basic business system and the heart of most organizations TPS is a computerized system that performs and records the daily routine transactions necessary to the conduct of business Primary goal is to automate computing intensive business transactions, such as those undertaken in the financial and human resource functional areas TRANSACTION PROCESSING SYSTEMS Inputs=transaction data like financial information, human resource data Processing=sorting, listing, updating, merging Outputs=bills, paychecks, orders, detailed lists and reports Transaction Processing Data Representation and Storage Sequential files ordered set of structurally similar records Indexed files 2 part structure: data + index Databases multiple entities (tables) linked by key-fields minimal redundancy Data Systems and Record Systems How are they Similar? They both… capture and store data organize digital data enforce standards for representation of data protect data from destruction (accidental or intentional). make data available for people to use (search, browse, retrieve, etc.) But are TPS Good Recordkeeping Systems? Transaction Processing Primary Goal: to automate some activity or business process. Transaction Processing and Records Recordkeeping may be the business process …or records may be the byproduct of the business process; but recordkeeping is not the primary purpose of the system …or records may be difficult or impossible to locate or retrieve …if they exist there at all. Transaction Processing Record-oriented business process General Ledger keep record of debits and credits cumulative history of transactions: account, object, amount, date… Academic Record (Transcript) keep record of courses, grades, degrees cumulative history of academic career: course department & title, instructor, grade semesters and dates of enrollment degree, school, major, honors, date Transaction Processing Records as byproduct Hotel Reservation retrieve room vacancy information collect guest information assign guest to room record guest information & reservation dates Employee Payroll combine pay rates and hours worked produce paychecks record annual totals of earnings, withholdings, benefits But managing complete records of all business transactions over the life cycle is not the primary objective of the system Transaction Processing Systems and Record Systems Why aren't TPS reliable Recordkeeping Systems? They capture incomplete information. Records are made up of content, context, and structure. Many Information Systems capture only content, and not always completely. They keep only current information. Many Information Systems store only "current values" for key bits of information and they do a bad job of tracking the history of changes to these values. Transaction Processing Systems and Record Systems Why aren't TPS reliable Recordkeeping Systems? They scatter information. To avoid redundancy, Information Systems store each bit of information just once, often in separate tables or databases, and they rely on computer software to re-connect the bits of information when needed. Related pieces of information become disconnected over time, or may be preserved or discarded according to different timetables. Transaction Processing Systems and Record Systems However…in many cases TPS ARE (de facto) Record Systems. An institution or agency may be using a TPS to conduct its business, and relying on the databases of that system to keep a record of key activities. It may not be a really good Record System, but that Data System IS the institution's (agency's) primary system for keeping records. Key issue: Is it possible to make an adequate Record System from a Transaction Processing System? DECISION SUPPORT SYSTEMS A computer-based information system that provides a flexible tool for analysis and help managers in making strategic decisions Uses data and information to produce flexible, on-demand reports or information, and assistance in decisions about unstructured problems By means of models, DSS can provide valid representations of real world systems DATA WAREHOUSE The data warehouse concept is about unbundling the two environments. In one environment – TPS - the business automates its processes on many different on-line transaction systems in the most effective and expedient manner possible. The data from these many and varied systems is then used to populate a database comprising all the data necessary to support decision making in a separate data warehouse environment. DATA WAREHOUSE Unlike operational databases that are set up to handle transactions and that are kept current as of the last transaction or update, data warehouses are analytical, subject oriented and are structured to aggregate transactions as a snapshot in time. DATA WAREHOUSES - PRIMARY CHARACTERISTICS Separate from the Operational systems and populated by data from these systems Available entirely for the task of making information available to users Time-stamped and associated with defined periods of time Subject Oriented as defined by the customer Accessible to users who have limited knowledge of computer systems Data Warehouses as Recordkeeping Systems Data Warehouses are NOT ERMS Do not meet many of the requirements 1) Do not routinely capture records – Evidence of Business Transactions – More concerned about transforming data into information 2) Do not manage information or records over the entire life cycle EDMS as Recordkeeping Systems Many EDMS functions overlap with ERMS EDMS functionality typically includes: indexing of documents, storage management, version control, integration with desktop applications, and retrieval tools to access the documents But they are not fully functioning ERMS EDMS As A Recordkeeping System EDMS: Primary purpose is to support day-to-day use of documents for ongoing business ERMS: Primary purpose is to provide a secure repository for authentic and reliable business records EDMS AS RECORDKEEPING SYSTEMS WHAT IS A DOCUMENT? “A grouping of formatted information objects regardless of medium or form that can be accessed and used by a person.” “A document is a container which brings together information from a variety of sources, in a number of formats, around a specific topic, to meet the needs of a particular individual.” A document is discrete and identifiable Structural unit of text like a report or letter or minutes of meetings EDMS AS RECORDKEEPING SYSTEM WHAT IS A RECORD? A specific type of information produced by a business event Evidence of business transactions Evidence is metadata documenting the context of creation, and the content and structure of the record Forms the basis for Recordkeeping Systems EDMS As A Recordkeeping System EDMS allows documents to be modified and exist in several versions ERMS prevents records from being modified EDMS As A Recordkeeping System EDMS may allow documents to be deleted by owner of data ERMS prevents records from being deleted except in certain strictly controlled circumstances EDMS As A Recordkeeping System EDMS may include some retention controls ERMS must include rigorous retention controls EDMS As A Recordkeeping System EDMS may include a document classification scheme ERMS must include a robust record classification scheme EDMS As A Recordkeeping System Most EDMS do not meet many of the requirements of a EDMS 1) Do not routinely capture records – Evidence of Business Transactions 2) Do not manage information or records over the entire life cycle Strategies for Capturing Record Content and Record Metadata Overall Goals of Record and Metadata Capture System officially “captures” records for the University Capture involves process of: 1) Registering a record 2) Deciding which class it should be classified to 3) Adding further metadata to it 4) Storing it in the ERMS. STRATEGIES FOR CAPTURING RECORDS 1) Applications that include BUSINESS PROCESS ENGINES 2) Applications that include WORKFLOW ENGINES 3) RECORDS MANAGEMENT APPLICATIONS (RMA) Business Process Models Primary Advantage: Models clearly identify all activities within a process and consequently identify all records and the relationships between records and processes It is a complete representation of the business process and of the various inputs and outputs generated by the process Business Process Models Primary Disadvantage These models are not often designed into the system automated business process engines are not the norm in North America Good Examples of automated business process engines are the European Registry Systems Registry System in Finland All recordkeeping functions in a Finnish government agency are supervised by a recordkeeping schedule Based on a hierarchical classification of agency functions, activities and business processes which serves the registry system filing appraisal Registry System - Operational Procedures Registration is based upon a business process e.g, filling an archivist’s position in the university archives This business process gets a registry number that is based on the functional classification schema uniquely identifies the business process in question links the business process to the function it serves, in this example personnel management Registry System - Operational Procedures Each business process is registered step by step regardless of the boundaries of the organizational units that participate in the process Records created by each step are filed in a document management system and are linked to registry entries Example of Recordkeeping in Registry System - Filling an archivist’s position in the University Archives Date Transaction Agent 2001-04-24 an announcement of the vacant position sent to a local newspaper an application sent by Ms. X an application sent by Ms. Y an application sent by Mr. W applications sent for review to the archives statement of the applicants sent to Office B after interviews a request for additional information from the archives a revision to a previous statement to Office B a decision to appoint Ms. Y as an archivist a letter to Ms. Y informing her that she has been appointed Office A R Office A Office A Office A Office A Archives R R R Office B R Archives Office B Archives R R R 2001-05-10 2001-05-10 2001-05-12 2001-05-24 2001-06-24 2001-06-30 2001-07-10 2001-07-25 2001-07-26 R Advantages of the Registry System For records and archives management the registry links records to business processes and functions that create them other records created by the same business process and function Provide contextual information and ensure the integrity of electronic records Conceptual Design – Workflow Workflow is "the automation of a business process, in whole or part, during which documents, information or tasks are passed from one participant [human or machine] to another for action, according to a set of procedural rules.” http://www.e-workflow.org/ “Starting from creation and ingestion, we should integrate the workflow process with the preservation process: appraisal, verification, maintenance and, eventually, retirement.” Su-Shing Chen “The Paradox of Digital Preservation” Computer (IEEE Computer Society), March 2001 WORKFLOW MODELS Advantages: 1) Commonly used 2) Coming back into fashion with an emphasis on life cycle management 3) Often Automated WORKFLOW MODELS Disadvantages 1) Primarily a Routing mechanism and thus: a) not all records that are created go through workflow process b) routing process may not identify all activities, all inputs and outputs within a given process; consequently some records within the process may not be identified and captured UIS EDEN Workflow Engine Overview of workflow engine for IU’s OneStart portal. OneStart & EDEN Component-Based Development OneStart Desktop Adaptable Personalized Customized User Interface Application Delivered Channels Other IUIE FIS SIS HRMS Applications Other Content Services Users Record Keeping Security Workflow Infrastructure EDEN Application Services Goals of EWE Service to enterprise applications that routes electronic transactions to individuals or systems for work, approval or notification. Audit trail of all routing and actions taken on electronic transactions. Integration with IU’s portal. Technical Architecture Component Based – EJB’s Documents processed in XML format Application interface Application post-processors Route Modules Minimal content is EWE route control data EWE can be used as the storage for pending transactions. EWE can enable versioning of information. Web services – applications required to register a web service for each document type. Document creation Application using EWE Applications send XML version of electronic document to EWE. (Document must be of a registered type.) Workflow engine Request route ID for doc Route Document Application using EWE EWE submits XML to route modules Workflow engine RouteDoc( XML contents of doc ) Discover Action Requests Routing Modules Workflow engine 1) XML for document 2) Action Requests 3) Add action requests to document route Route modules return list of action requests Route modules scan XML for specific route controls and match document information to action request rules. Activate Next Action Request Document’s action requests are activated by: 1. 2. 3. Route type priority as determined by document type route template Request priority as determined by the route rule Order in which request was associated with the document. The activated requests are put on the appropriate action lists of the persons or processes. Review Action List Person reviews action list in the portal. Person is able to see the routing log and other header information. Open document Action list refers user to original application to see any business information in the document. Application consults with engine to determine what elements of the document to show the user, if any. Approve Document User acts upon the action request. Application business rules are applied to control actions taken on the document. Route Document again Application using EWE Workflow engine RouteDoc( updated XML contents of doc and action taken by user ) Post-process document Once the engine has determined that all routing is complete, it then notifies the application post-processor. This is an EJB for the application that does appropriate work with the document such as changing pending status, updating calculations, generate new documents, etc. ADDITIONAL FEATURES Add Recordkeeping Routing Rules Add Recordkeeping Environment Conceptual Design – Workflow and Electronic Recordkeeping Applications EDEN Portal (Infrastructure) FIS (User Interface) Inbox HRMS OneStart Workflow Engine Purchasing Preference Engine Recordkeeping RECORDS MANAGEMENT APPLICATIONS (RMA) TRIM Attempts to include the functionality of: EDMS to provide integration with desktop applications, add version control, and collect key documents Records Management System to file records within the framework of the record management architecture and to manage records over their life cycle Knowledge Management to build relationships between objects to enhance retrieval Workflow to combine processes and objects TRIM TRIM manages and integrates both electronic and physical records TRIM supports the capture and import of e-mail messages and their associated attachments TRIM Control Mechanisms: Registration – Provide evidence that a record has been created or captured in a recordkeeping system Classification: TRIM has a Record Plan or File Classification module that allows organizations to build a classification system that reflects the business processes TRIM also has a concept of Record Types to allow for further classification TRIM Retention and Disposition: Classification scheme is linked to Disposal Schedules TRIM allows organizations to import disposal schedules or to create their own Schedules are attached to records to allow for automated disposal or retention TRIM Thesaurus: TRIM has a Thesaurus module that conforms to the ISO standard TRIM allows organizations to build a thesaurus or to import thesaurus terms for an accepted thesaurus TRIM Access control is provided through a combination of individual “Administration Access” rights, view and edit privileges assigned via the record plan, and system option setup assignments. TRIM Migration and Conversion TRIM provides the tools to migrate records to another relational database system TRIM Tracking: Audit Trails TRIM offers the capability to perform two levels of audit logging. Core logging includes logging of the record title, number and container changes, record movements, record deletions, and all electronic activity. Full logging captures the elements of core logging plus record creations, changes to security, notes, retention schedules and triggers, dispositions, thesaurus terms, record plans, location security, location deletions, and workflow processing. Physical Tracking of Paper Records: Barcoding, Location Designations, Movement History TRIM TRIM provides an Application Programming Interface (API) toolkit to allow integration with many other business system applications in an effort to provide an integrated environment TRIM TRIM 4.3 was tested and certified as complaint with DoD 5015.2 Standard in 1999 and again in 2001. Statement from 1999 DoD compliance test: TRIM “integrates electronic document management and records management and provides a single interface to manage organizational records, including electronic and nonelectronic records.” http://jitc.fhu.disa.mil/recmgt/#standard OTHER STAND ALONE RMA PRODUCTS ForeMost Enterprise, Version 2 by TrueArc, Inc. Tarian eRecords Engine v1.0 - formerly eRecords v1.0 by Tarian Software, Inc. iRIMS 2001 by Open Text Corp. OBJECTIVE 2000 by Objective Corp. Hummingbird RM Family 4.0 by Hummingbird, LTD FileSurf 7.0 by MDY Advanced Technologies, Inc. R2M (Relativity Records Manager) Version 2.0 by Relativity, Inc. EDMS product that adds Records Management Functionality R2M incorporates records management seamlessly into a document management environment. The environment is open structured and can be presented as defined by the document and records managers. All documents, including records, must be brought into the document management environment prior to any other activity being taken upon them. As implemented, a record in R2M is simply a specially managed document. R2M extends the concept of the document lifecycle to include record dispositions. eManage 2000 by ByteQuest Technologies, Inc. Knowledge Management product that adds Records Management Functionality eManage incorporates records management seamlessly into its knowledge management environment. The environment is presented as a file structure or file tree. The records manager and systems administrator work together to develop and implement a file plan as a branch or branches of the organization's knowledge file tree. Integrated RMA Products STAR/RIMS-E by Cuadra Associates, Inc. Integrated product using two products from the same company STAR is a network database management system environment in which Cuadra has developed many other information and archives management applications. STAR/RIMS-E is a customizable application developed by Cuadra Associates, Inc. in their STAR environment. The system is very flexible and can address large, complex Records Management problems. The system supports the management of large stores of paper and non-electronic documents, as well as e-mail and other electronic documents -all within the same environment. Integration of FileNET IDM Content Services 5.1.1 and ForeMost Enterprise 2.0 FileNET/ForeMost is an integrated product that combines the document management capabilities of FileNET IDM Content Services with the records management capabilities of ForeMost Enterprise. ForeMost provides the records management functionality for the pairing and uses the FileNET repository for storing records filed from FileNET. IBM e-Records Solution (IeRS) version 1.0 by IBM Corporation IBM’s IeRS is a combination of IBM’s Content Manager v7.1 and Tarian Software’s Tarian e-Records (TeR) v1.0. IBM's Content Manager provides document management and workflow capabilities. It also provides the declare, search, retrieve functions, and records repository for this solution. TeR v1.0 is a web-based RMA and provides the records management and access portions of the solution. Work Manager/ForeMost by Eastman Software and TrueArc, Inc. Integrated Product Work Manager/ForeMost is an integrated product that combines the document management, imaging and workflow capabilities of the Work Manager Suite with the records management capabilities of ForeMost Enterprise. The result is a system that stores and manages both official records and their predecessor “draft” documents to provide appropriate control over the full lifecycle of the items, from origination to disposition. OmniTREEV v2.3 by TREEV, Inc. with ForeMost Enterprise Version 1.0 by TrueArc OmniTREEV/ForeMost is an integrated product that combines the document management, imaging, and workflow capabilities of the OmniTREEV environment with the records management capabilities of ForeMost Enterprise. iRIMS v7.1.7 by Open Text Corporation with DocuPACT 2000 by InterTech In the integrated product, iRIMS provides the specialized records management functionality, while DocuPACT provides the NT File System-based electronic document/record repository, storage of document/record metadata in a relational database, and document/record search capabilities.