1. Evaluating Relational Database Management
Systems Using the OSDL DBT Workload Module
1.1 Purpose
Following the last fiscal year, we have been developing evaluation methods that are
required in the choosing of open source relational database management systems
(RDBMS) for use in corporate information systems.
A database is the core of the comprehensive management of enterprise data. Many
enterprise information systems are built upon database technology.
Open source products are also experiencing a shift of developers’ attention from the
OS level to the middleware level. Open source RDBMS is becoming the center of
interest.
Under such circumstances, there is a growing need for a performance evaluation
method that serves as a criterion for system developers to choose RDBMS products,
which follows that having a standard evaluation procedure is significant for
determining a suitable range of applications for open source RDBMS.
Our research is intended to respond to such demand. We modified an existing
RDBMS performance evaluation tool to make it more versatile and then defined the
procedures for building an evaluation environment and performing evaluation that are
required for system development. Finally, we verified that our procedures serve as a
standard for sizing and tuning real-world RDBMS systems. As target systems, we
chose 32-bit and 64-bit CPUs. The latter is an extension of the IA32 architecture,
recently receiving attention. We also compared the performance between the two
systems. We applied our performance evaluation procedures to major open source
RDBMS. The results will be presented for reference in a later part of this document.
We used the OSDL DBT series product as a database workload tool. The DBT tool is
provided under an open source license. It is a reasonably-priced easy-to-use RDBMS
evaluation tool. We chose this tool based on the following criteria.
(1) The tool must have a DB benchmarking function (i.e., provide a close-to-real
environment).
(2) The testing environment must be easy to build (i.e., in cost and implementation
effort).
(3) There must be few limitations in operation (i.e., in functions and cost).
(4) There must be few restrictions in publication of the evaluation result.
(5) The tool must serve many purposes (i.e., able to handle open source RDBMS
and various system sizes).
− 1-1 −
DBT is an open source workload tool for simple TPC performance evaluation. We
think it fits our purpose because it is free and the developer imposes few restrictions
on publication of evaluation results.
Our aim is to provide a guideline for making the utmost use of open source RDBMS
by defining performance evaluation procedures and giving general hints for applying
the procedures. We do not intend to compare specific hardware and software
products.
1.2 Project Scope
We chose some popular open source RDBMS products and performance evaluation
tools we thought to be valuable.
 For OSDL DBL-1 (hereinafter called DBT-1), we converted the stored procedures
that depend on individual DBMS into C-language coding to enhance portability
and versatility.
 For the target open source RDBMS for which the evaluation procedures are
defined, we chose PostgreSQL and MySQL.
 For the application models run in the 32-bit and 64-bit (EM64T was chosen this
time) environments, we used DBT-1, a Web transaction application used in the
last fiscal year, and OSDL DBT-3 (hereinafter called DBT-3), a decision-making
system application.
Table 1.2-1 compares the project scopes of this and last fiscal years.
Table 1.2-1: Project Scope
Second Half of Fiscal Year 2004
DBT-1
DBT-3
First Half of Fiscal Year 2005
Evaluation of the following
Porting to MySQL (ODBC)/
versions:
PostgreSQL (ODBC) and evaluation
MaxDB (Stored)
Evaluation of MaxDB (Stored)
PostgreSQL (Stored)
version in 32- and 64-bit
Oracle (Stored)
environments
Evaluation of PostgreSQL
Evaluation of PostgreSQL (Stored)
version in 32- and 64-bit
environments
− 1-2 −
1.3 Overview
1.3.1 Evaluation Criteria
We defined the following two criteria for this evaluation.
1.3.1.1 Enhancing Portability and Expanding Application Range of DBT-1
(1) Modification and Evaluation for MySQL
We modified DBT-1 for use with MySQL based on the research on the results of
modification in fiscal year 2004. (Modification includes conversion from executing
stored procedures to issuing SQL statements.) We also evaluated MySQL using
the modified DBT-1 in the following environment.
 Database and related products: MySQL 5.0
 Operating System: MIRACLE LINUX V3.0
 Performance Evaluation Tool: OSDL DBT-1 MySQL (ODBC) version
(2) Modification and Evaluation for PostgreSQL
We modified parts of the source code created in (1) for use in PostgreSQL and
confirmed that DBT-1 operation does not depend on stored procedures. We also
evaluated PostgreSQL using a modified DBT-1 in the following environment.
 Database and Related Products: PostgreSQL 8.0, 8.1
 Operating System: MIRACLE LINUX V3.0
 Performance Evaluation Tool: OSDL DBT-1 PostgreSQL (ODBC) version
1.3.1.2 Performance Evaluation in 64-bit Environment
(1) Performance and Reliability Evaluation of 32- and 64-bit Environments Using
DBT-1
We evaluated performance with the 32- and 64-bit architectures in the following
environments using DBT-1 for MaxDB, which is the result of research in fiscal
year 2004.
 Database and related products: MaxDB 7.5
 Operating System: MIRACLE LINUX V3.0 and V3.0 for x86-64
 Performance Evaluation Tool: OSDL DBT-1 MaxDB (Stored) version
− 1-3 −
(2) Performance and Reliability Evaluation of 32- and 64-bit Environments Using
DBT-3
We evaluated performance with the 32- and 64-bit architectures in the following
environments using DBT-3 for PostgreSQL, which is the result of research in
fiscal year 2004.
 Database and Related Products: PostgreSQL 7.4, 8.0
 Operating System: Red Hat Enterprise Linux AS release 4
 Performance Evaluation Tool: OSDL DBT-3
− 1-4 −