MPRIME Preliminary Architecture July 11, 2002 (Draft #3) Background The Michigan Program for Research Information Management and Education (MPRIME) is a project, which aims to improve the protection of human subjects in research by improving the administration and approval process for projects and protocols involving human subjects. In April 2003, MPRIME issued an RFP for products which support core processes that impact human subject research compliance. In July 2003, Webridge Extranet Portal emerged as the leading product in the RFP selection process. This document provides a high-level overview of the technical architecture that would be needed to support a Webridge deployment. Details of the architecture are likely to change during implementation, but this document represents a likely scenario of how the architecture will be deployed. Webridge architecture summary More than any product that we have seen to date, the Webridge software architecture is based completely on Microsoft standards. Company representatives proudly state that they have strategically aligned their technology with Microsoft, and believe this alignment has proven to be wise and has contributed to their corporate success. The Webridge product is primarily web-based, and is based on a classic n-tier architecture. The components of the architecture include: The Browser All end-users, administrators and developers will use a web browser to interact with the Webridge applications. Like most web applications, the browser primarily handles presentation of the application. No business logic occurs on the client. Some Webridge development cannot be done through a browser, and must be completed using Webridge “fat client” tools. U-M anticipates that this type of development is rare, and that the vast majority of development will be done through a web-interface. Web Server Microsoft Internet Information Server (IIS) is used for all web communication. IIS’ primary role is to deliver and receive HTTP/HTTPS traffic to and from the client. Very little Webridge processing occurs within IIS. The exception is an Active Server Page (ASP) that is used to route requests properly to the application server. Application Server The Webridge application server is the heart of the Webridge system. The application server is responsible for nearly all Webridge business logic. Responsibilities of the application server include: Retrieving business logic from the database, and then executing all business logic. Sending all query and update requests to the database. Construction and deconstruction of objects in memory through the use of XML and database calls. Developing HTML for presentation of to end-user Management of memory and disk cache. The application server is build with Microsoft Transaction Server. Originally written in Java, the Webridge application server is now been ported to “J#” (“J sharp), Microsoft’s modified version of Java. The Page 1 of 16 application leverages many of the core services that are available on the Microsoft Windows Operating System. Database Server Webridge relies on Microsoft SQLServer for all database services. Not only is traditional application data stored in the database, but also much of the application business logic is also stored in the database. This is similar to the way PeopleSoft stores the business logic within “PeopleTools” tables. All business logic is stored within the database in XML format. The application in turn can then retrieve the XML, and construct objects as necessary. Webridge also takes advantage of SQLServer stored procedures. Stored procedures process some business logic, and are used to support other interfaces within Webridge, most notably the interface between the Corpus and the database. The stored procedures can be modified by the U-M, but Webridge technical representatives discouraged this. File Server A significant amount of the application content in Webridge is stored within traditional document files (e.g. Microsoft Word, Adobe Acrobat). Document files can be stored on any Microsoft file system. Optionally, a file server can be added to the environment to handle file storage and retrieval. Webridge refers to the repository of documents as the “Corpus.” The Corpus is a simple set of directories for storing files, which can be navigated by the Webridge application server to retrieve files. Index Server Webridge applications allow end-users to do search structured (database tables) and non-structured (documents) data. Non-structured searches are handled through the use of the Microsoft Index Server. The MS Index Server maintains a file-based index of all documents within the Corpus. This allows Webridge to support complex searches by end-users. Customization and configuration Webridge is an extensible environment. Webridge customers can use Webridge-provided tools to modify various components of Webridge applications. Webridge classifies modifications in two ways: Configurations Through the use of Webridge web-based tools, developers can modify aspects of the Webridge application, such as web forms and work flows. Webridge uses the term “configuration” to describe these modifications, because these types of changes are non-programmatic in nature. The implication of the term “configuration” is that the modification is not very significant to the application. Unfortunately, this is misleading because some modifications can have significant impact. For example, some configurations can result in changes to the underlying database structures, including adding new data types. Configurations are “protected” during upgrades to the Webridge application framework. Extensions to the database structure are rolled over untouched to the next upgrade. The web-based tools does allow scripting. Scripting can be used to verify data, call external code, etc. Scripts are not “protected” during upgrades. Scripts would need to be thoroughly verified during upgrades. Customizations Through the use of Webridge “fat client” tools, Site Designer and Entity Manager, developers can modify various definitions that are stored within the database as XML definitions, and stored procedures. Examples of common types of customization include the integration with outside systems and the definition of complex nested reports.. Also if the SQL generated by Webridge is inefficient, it is possible for U-M staff to add “hints” inside of stored procedures. The tools allow programmers to modify any data type, including the “base data types” in the system. Customizations to base data types are not protected during upgrades of the Webridge product and need to be tightly managed if they are undertaken. Ideally, U-M would not modify these base data types, but define related data types to store required information. Page 2 of 16 MPRIME databases requirements An MPRIME implementation of Webridge would likely involve the following environments, each with their own SQLServer database: Production This would be the primary Online Transaction Processing (OLTP) database for MPRIME. Mission critical University data would be stored in this environment. Reporting A read-only database created by a Webridge-provided batch conversion program. While this environment is “data warehouse-like,” it should not be considered the data warehouse. It can be better described as a mix between a data warehouse and an operational data store (ODS). It’s not quite as well planned as the U-M’s data warehouse, but it’s not an exact replica of production like the ODS. The reporting data only contains structured data in the database. The Corpus (files) is not copied to the reporting database. Quality Assurance (QA) A staging area, where customizations and configurations are tested before they are moved to the Production and Reporting environments. At this point, the presumption is that this will be a productionsized environment which is replicated from the Production database a few times each year. Development (multiple) An environment used by a developer to customize or configure the Webridge applications. Each developer has his/her own development environment. Development environments are stored locally on the developer’s workstation. It is important to note that the preceding list does not include many types of databases, which are in use, or have been requested for other systems. For example: demonstration, stress testing, training, upgrade and user “playground” will not be provided as part of MPRIME. End-user desktops Webridge supports both Internet Explorer (4 and greater) and Netscape (4.78 and greater). According to the vendor, the product is based on HTML standards and therefore should not pose problems on different browsers and operating systems. The vendor did indicate one area where cross-platform compatibility is not available: users that wish to transfer reports directly into Microsoft Excel can only do this on Internet Explorer and Windows computers. This functionality will not work with Netscape or the Macintosh operating system. Developer desktops Browser-based development (configuration) can occur on the same platforms supported for end-users. Fat-client-based development (customization) can only occur on Windows computers with the complete set of Microsoft software (IIS, MTS, SQLServer). It is also possible to provide the fat-client tools from Citrix. During implementation we may decide this is a more efficient and effective model for distributing these tools. Page 3 of 16 Architecture design points In the following pages, the hardware and network architecture for the Webridge system is detailed. The design of this architecture is based upon a number of fundamental decisions and assumptions. These include: 1. The ITS storage area network (SAN) should be used for all database and file storage (corpus). Availability, performance, recovery and replication requirements are likely to require the robust SAN architecture. 2. Any ITS server, which is connected to the SAN, should be in the ITS Secure Zone. To maintain the security around the SAN network, access to the SAN is limited only to machines that are in the ITS Secure Zone. This is a ITS standard that could only be changed through discussion with the ITS Information Security Officer and the ITS Associate Vice President. 3. The web and application server need to reside on the same physical machine. Webridge technical staff indicated web and application servers could reside on different servers, but that they only support configurations where they are on the same server. 4. Production and Reporting databases will reside on the same server. Reporting database will only used by a small set of people at a time, and probably does not warrant its own complete set of hardware. (Note: The index server and index files need to be on the IIS/Application server.) 5. All web servers should reside in the ITS DMZ. This is also a ITS standard. Since the purpose of web servers is to interact with users outside of the firewall, the DMZ is the appropriate location for web servers. Note, the combination of #1, #2, #3 and #4 above means that every MPRIME environment (e.g. Production, Quality Assurance) will require two servers: a web/application server in the DMZ, and file/database server in the Secure Zone. 6. Production and Reporting database instances can share one web/application server. Webridge technical representatives indicated that it is common to see multiple databases connected to one web/application server. The benefits of sharing a web/application server include: fewer migrations, fewer upgrades, and fewer physical boxes to administer. The disadvantages of sharing a web/application server include: system competing for processing resources and the systems are tied together for all upgrades and maintenance. 7. Redundancy of servers will not be pursued at this time. For example, there is only one web server, one application server, and one database server. We will purchase a hot spare that we can use in the event of a massive hardware failure. The hot spare server will also be used as an upgrade server to test new Webridge releases without impacting the running Production, QA, and Development environments. 8. No load balancing is needed, but SSL acceleration should be used. Even though there will be only one web server, ITS’ will use its load balancing infrastructure to provide SSL acceleration. This will offload processing from the MPRIME web server and help with performance. Other than SSL acceleration, the load balancing infrastructure will not be leveraged. 9. No training environment will be necessary. Will use test for training purposes, limited hands on training required for application. 10. No stress-testing and application tuning environment will be necessary. (Judy to confirm.) Since little application tuning can be done inside of MPRIME, a separate “stress” database will not be needed. Tuners will have to use the Quality Assurance environment to tune applications. Note, this could Page 4 of 16 result in slower development schedules because developers and tuners will need to share common objects, and therefore may not be able to work as efficiently. 11. Cosign will be used for authentication. Technical representatives indicated that authentication exits are available which will allow the U-M to implement Cosign. U-M has already developed the necessary Cosign code to work with Microsoft IIS applications, so we believe integration of Cosign with Webridge should be very straight-forward. (What about NDS for Medical School.) 12. No Citrix or Window Terminal Services will be needed. All client and developer tools will be implemented through browsers and fat client installations. (Or should the two fat clients be implemented on Window Terminal Services?) 13. MPRIME servers will be placed within existing M-Pathways production windows domain. Sharing the M-Pathways domain will be more efficient, because more servers won’t have to be purchased and administered to support the domain. Putting the Development/Quality Assurance servers in this domain makes sense because some developers will not be ITS staff, so the ITS domain wouldn’t make as much sense. 14. No central printing or online viewing will be needed. All reports will be delivered through the Webridge interface, therefore no interfaces to Dazel and Vista Plus will be needed. 15. No Electronic Data Interchange (EDI) will be needed. There are no EDI interfaces with the Webridge system. 16. All interfaces to MPRIME are file-based. All interfaces between MPRIME and other systems will be based upon the exchange of files. For example, data needed from the PeopleSoft systems will be exchanged by files through standard mechanisms (SSH). (Make this assumption. We will not know this for sure until we get into detailed design, but using our current model , we believe this to be true) 17. No Virtual Private Networks (VPN) will be needed for interfaces. Since all interfaces are within the U-M, no VPN will be needed to support secure transmission of interface files. 18. Virtual Private Networks (VPN) may be needed to support developers outside of the ITS network. It may be necessary for developers to share common file systems or other services. Also, some developers need to have direct DCOM communication to the application server. In order to assure these transmissions are secure, VPNs may need to be deployed. 19. Customizations will be managed through Microsoft Visual SourceSafe (VSS) software. Developers will check-in, check-out and manage all objects through VSS. VSS requires file server access to developer desktops. 20. Configurations will be managed through a manual process. No tools exist to manage migrations of configurations. Webridge provides a method to export and import many types of changes from one environment to another, but this process is not automated. The tools allow for a “package” to be created, but then the packages are applied to each environment manually. MPRIME staff will need to have a formal procedure for migrating these changes. Note, some development objects cannot be migrated and need to be reapplied by hand in each environment. (Cindy will research.) 21. No development will occur on the production server. Page 5 of 16 Webridge technical staff described methods of using the production server for development. It involved saving changes in a holding area before applying them to production. This approach will not be used, because it will make synchronization with Quality Assurance and Development too difficult. Page 6 of 16 Preliminary Hardware Plan The following diagrams depict the hardware that will be used by MPRIME, and the software components which will reside on each piece of hardware. Black circles (with numbers) indicate new servers which need to be purchased to support MPRIME implementation. All other equipment is already in production within ITS. Diagram #1: MPRIME Production Production M-Pathways Backup Domain Controller Production M-Pathways Primary Domain Controller Microsoft Internet Explorer Microsoft Office Applications 1 Production Web/ Application Server User Webridge "Pass Through" Web Transaction (ASP) https Web Server (MS IIS) https Webridge Application Server (MTS) Production DB/File Server Oledb Production (SQLServer) UMich Cosign ISAPI Extention MS Index Server Index of Corpus Content Batch Program To Create DB Stored Procs oledb Webridge ISAPI Extension 2 SMB /Net BIO SP orts 135 Reporting (SQLServer) -1 3 9 Production Webridge Corpus tcp Production ITCS Server UMich Cosign Server The production MPRIME environment will be made up two servers: production web/application Server and the production database/file Server. Not only does the database/file server house the production database, it also houses the reporting database as well. The MPRIME servers need to operate within a standard Windows domain. Instead of creating a new MPRIME domain with the overhead of additional equipment and management, MPRIME will be placed within the existing M-Pathways production domain. The “domain controller” servers near the top of the diagram represent some of the key existing equipment from the M-Pathways domain that will be used by MPRIME. During the second half of 2003, ITCS will place a Cosign server within the ITS machine room. MPRIME will use this server as its primary authentication server. Note, this diagram does not include all servers that will be used (e.g. Kerberos, DNS, Unicenter, etc.), but instead includes only the primary servers that will make up the MPRIME environment. Page 7 of 16 Diagram #2: MPRIME Development & Quality Assurance Production M-Pathways Primary Domain Controller Microsoft Internet Explorer Webridge "Pass Through" Web Transaction (ASP) https Microsoft Office Applications Web Server (MS IIS) Webridge ISAPI Extension https Webridge Application Server (MTS) Webridge Site Designer oledb 4 QA DB (SQLServer) Dev DB Stored (SQLServer) Procs Stored Procs Index Server SMB/Net BIOS P orts Index of Dev Corpus Content Index of QA Corpus Content Specialized Developer Desktop Development & QA DB Server 3 Development & QA Web/ Application Server Standard Developer / Admin Desktop Production M-Pathways Backup Domain Controller 135-139 Dev Corpus (MS File Server) QA Corpus (MS File Server) UMich Cosign ISAPI Extention tcp dcom Production ITCS Server Webridge Entity Manager dcom UMich Cosign Server The development and quality assurance environment is identical to the production environment but with these relatively small changes: The two MPRIME servers will be less powerful (probably only half of the CPUs of the production servers). Some developers will connect to the application server using Webridge fat-client development tools over dcom. Separate corpuses must exist for development and quality assurance. (Cindy, we need to confirm that this really works. I believe that this does work.) Page 8 of 16 Diagram #3: MPRIME Network ht tp MAIS Machine Room Firewall ps U-M Network s DMZ Network Switch Specialized Developer Desktop User Desktop Standard Developer/Admin Desktop Specialized Developer Desktop dc om on Standard Developer/Admin Desktop htt aV PN User Desktop https Internet MAIS Staff Network Firewall Secure Zone Network Switch ITCS Production Cosign Server Production Database/File Server Load Balancing (SSL Acceleration) Test Database/File Server Production Web/Application Server https om dc MAIS Staff Network MAIS Storage Area Network Test Web/Application Server Demonstration Web/App/DB/File Server The network design for MPRIME is similar to other applications that are currently run by ITS with a few exceptions. These exceptions include: Fat client software will be deployed to some non-ITS desktops. This client software will require file server access and communicate using dcom. The use of a VPN to secure the connection may be necessary. Load balancing switch will only be used for SSL acceleration not for load balancing across multiple web servers. There will be some users of the systems that are not on the U-M network. Page 9 of 16 Version control architecture In order to meet its business requirements, the U-M will customize and configure various portions of the Webridge applications. The following diagrams illustrate how different types of objects will need to get migrated through each MPRIME environment. An important assumption in these processes is that each developer has his/her own local development environment. Multiple development databases create logistical problems, which require thorough management of the customization process. Diagram #4: Migration Path for Site Designer and Entity Manager Customizations Production Environment SD & EM Customizztions (Zipped File) QA Environment SourceSafe Export Webridge Site Designer Development Environment SourceSafe Repository Webridge Entity Manager Version control for Webridge customizations is handled through Microsoft Visual SourceSafe (VSS). This is a full-featured version control product with many advanced features. The following steps describe a likely process that would be used to develop a customization with Site Designer and Entity Manager and move them to production. 1. Developer “checks out” desired objects from the VSS repository. The objects are locked, so other developers cannot modify them until they are checked back in. 2. Developer uses Site Designer and Entity Manager to build and test customizations. 3. Developer moves customizations back into VSS repository. 4. Developer (or Migrator) exports changes to a “zip” file. 5. Developer (or Migrator) uses Webridge tools to apply the “zip” file to the Test server. 6. Testing is done and process is repeated until code is ready for production. 7. Developer (or Migrator) applies the “zip” file (or files) to the production environment. 8. Developer (or Migrator) unlocks the objects for use by other developers. Page 10 of 16 Diagram #5: MPRIME Configuration Migration Path Webridge Browser-based Configuration Tools Production Database Webridge Import/Export Utility Webridge Browser-based Configuration Tools QA Database Webridge Browser-based Configuration Tools Development Database SourceSafe Repository The Webridge product does have some tools to allow export and import of changes, but there is no advanced version control capability available. We plan to leverage VSS for this purpose and use it to be the keeper of the official copy of the application. The following process describes the steps that would likely be used to migrate configurations. 1. Checkout the project type definition from version control. If not using version control, the it is assumed that the production store is the keeper of the official copy and you should export the project type from the production server. 2. Import the project type definition into a development store 3. Make the required changed to the workflow via the web based administration forms and perform the initial testing on that store. 4. When satisfied that the changes are complete and accurate, export the project type definition from the development store. 5. Import the new project type definition into a Staging/Test server which is created from a recent backup of the production store. 6. Test the new version of the Project Type. If issues are discovered, return to step 3. 7. When satisfied that the changes behave as intended, backup the production store then take the same project type definition export that was imported onto staging and import it into production. It is recommended that this is done during a period of inactivity on the production server as, depending upon the nature of the changes, there may be a requirement to migrate data structures. 8. Check in the project type definition export into version control so that it can serve as the master for the next round of changes. Page 11 of 16 Diagram #6: MPRIME Stored Procedure Migration Path Production Database QA Database Script which modified Stored Procudure Development Database If stored procedure changes are related to the Site Design and Entity Manager, then migration process is the same as in Diagram #4. If the stored procedure changes are manual, such as tuning, then the following process would most likely be used: 1. Tuner builds script to make changes in the development environment. 2. Tuner (or Migrator) applies script against Quality Assurance environment. 3. Tuner (or Migrator) applies script against Production environment. These changes need to be carefully tracked, because any other changes to the stored procedures will write over these modifications, so they will need to be reapplied. Page 12 of 16 Implementation Challenges In developing this preliminary architecture document, several challenges were identified which need to be addressed during implementation. Webridge specific challenges New version control system ITS systems are currently based on version control using Quest Stat software. MPRIME will need to rely on a completely different version control software (VSS) and migration process. Relying on multiple products and processes could result in some inefficiency and overhead in the organization. Vulnerability of IIS, may adversely affect service level goals Microsoft IIS has proven to be the single biggest target for security attacks over the past few years. ITS has specifically avoided using IIS for enterprise systems for this reason. When IIS had to be used, ITS has set firewall policies to limit access to U-M sites, which prevented attacks that originated from other Internet sites. If security attacks continue to be an ongoing problem with IIS, ITS may find it difficult to assure the high service levels that are needed or desired for MPRIME. Poor scaling options Most applications today allow for the use of multiple web and application servers. Organizations can add as many servers as necessary to meet performance targets. Webridge only supports a single web/application server. Other than buying the biggest server possible, there is no way for hardware to be acquired to help address performance problems. Future of Macintosh support questionable Webridge is firmly and closely aligned with Microsoft technologies. While it appears the current version of Webridge can work with the Macintosh, there will always be a risk that future versions will diverge from the Macintosh when and if Microsoft architectures become less standards-based. Before proceeding with the purchase of Webridge, the U-M should recognize the distinct possibility that other operating systems (e.g. Macintosh, Linux) may not work with future versions of Webridge. No significant upgrade support tools There are no upgrade support tools to assist an organization assess the impact of a new version on their modifications. It will probably be necessary to have an accurate and thorough record of all customizations to help with the analysis during upgrade planning. The magnitude of this challenge depends on the scope of changes made by U-M (more changes, more risk), and if Webridge upgrades prove to be implemented as Webridge staff have indicated (if they misrepresented the situation, more risk). Little control over generated SQL The Webridge application server constructs almost all SQL on the fly, or from stored procedures. There is little opportunity for application tuning. If Webridge generates inefficient SQL, U-M’s only course may prove to be to inform Webridge and hope that they provide a patch. Solution for home development needed Since production support often requires developers to work from home or remote sites, it may be necessary to employ Citrix or other technologies to provide access to tools, file systems and other resources. (Note, at the present time we are assuming that no Citrix access is needed.) Page 13 of 16 MPRIME specific challenges Distributed development Current MPRIME plans call for developers to exist in multiple University units (at least ITS, DRDA and Medical School). Distributed development has proven difficult in the past and was abandoned on the M-Pathways project. Well-defined processes, which assure adequate controls, will be necessary and are likely to conflict with the approaches used in some University units. File formats Submission and review of document files is a key component of the MPRIME system. Sharing documents between different units introduces software compatibility issues. For example, a document may be stored in a version of Word or Acrobat which is available in one unit, but not available in another unit. The problem becomes exaggerated over time, as files stored in the system become many years old, there may not be software available to open them. No data growth or sizing metrics available No data growth or sizing information is available about MPRIME and Webridge at the present time. Therefore hardware configuration and costing estimates are just guesses with little or no factual basis. 7x24 Support not available ITS backup and recovery architecture, and maintenance requirements prevent ITS from providing 7x24 support. ITS could commit to 7x24 with the exception of Sunday morning form midnight-noon. This window would be needed for backups and maintenance. Note, some additional planning is needed to assure backups can be fit into this window (due to other backups at the same time period). Novell authentication may need to be supported Faculty at the U-M Medical Center tend to be more familiar with their Novell authentication (NDS) than their Umich password. The Medical Center would like to see two types of authentication available for MPRIME. It is not clear if this is technically possible, or if it is, if it is wise to do. Logical servers may be better than physical servers Instead of purchasing several smaller servers, it may be better to by larger more powerful servers and use logical partitioning to create logical servers. While this may prove a wise strategic direction, funding issues could make this difficult to address. Page 14 of 16 Appendix 1: MPRIME Preliminary Hardware Configuration and Pricing Information One Time Costs Component Name Description Projected Cost New Server #1 Production Web and Application Server 4-CPU Windows Server with X GB of RAM 3x36 (RAID5) direct-attached disk $20,000 New Server #2 Production Database & File Server 4-CPU Windows Server with X GB of RAM 2x18 (mirrored) direct-attached disk SAN Adapter $23,000 New Server #3 Development & Quality Assurance Web and Application Server 2-CPU Windows Server with X GB of RAM 3x36 (RAID5) direct-attached disk $12,000 New Server #4 Development & Quality Assurance Database & File Server 2-CPU Windows Server with X GB of RAM 2x18 (mirrored) direct-attached disk SAN Adapter $15,000 New Server #5 Hot Spare/Upgrade Web, Application, Database & File Server 2-CPU Windows Server with X GB of RAM 3x36 (RAID5) direct-attached disk SAN Adapter $15,000 Prod/Report Disk 200GB at $50 per GB $10,000 Dev/QA SAN Disk 200 GB at $50 per GB $10,000 Subtotal $110,000 Ongoing Costs Description Projected Cost Server hardware maintenance (15% of purchase price) Includes SAN $16,500 Server hardware replacement (20% of purchase price) Includes SAN $22,000 Disaster recovery hot site equipment Subtotal $1080 $39,580 Page 15 of 16 Appendix 2: MPRIME Preliminary Software Configuration and Pricing Information One Time Costs Component Name Description Projected Cost CA Unicenter Monitoring and job scheduling Needed on production database/file server Tivoli Storage Mgr Backup and recovery Needed on all four new servers $3,850 MS Windows OS Operating System Needed on all four new servers Included in Hardware Price $0 MS Internet Information Server (IIS) Web Server Needed on three servers, and all developer & migrator workstations Included in Hardware Price $0 MS Transaction Services Transaction Server Needed on three servers, and all develop & migrator workstations. Included in Hardware Price $0 MS SQLServer Database software Needed on two database servers. Included in Hardware Price $0 MS Visual SourceSafe Version control software. Free from Microsoft $0 $13,000 Subtotal $16,850 Ongoing Costs Description Projected Cost CA Unicenter maintenance $3,830 Tivoli Storage Mgr maintenance $1,400 MS Windows OS maintenance $ MS Internet Information Server (IIS) maintenance $ MS Transaction Services maintenance $ MS SQLServer maintenance $ MS Visual SourceSafe maintenance $ Subtotal $5230 Page 16 of 16