The Open Source Database Benchmark (original) (raw)

Index
Home
FAQ
Run Rules
Building OSDB
Style Guide
Project Status

OSDB Links
-Get OSDB here!
-The CVS Repository has the latest (maybe broken) sources
-OSDB mailing lists
-Contact us

Other Links
-Linux Benchmark Suite
- Association Française des Utilisateurs d'Unix et des Systèmes Ouverts SSBA benchmark suite

OSDB has tests underway for
Informix
MySQL
PostgreSQL

OSDB needs tests for
FirebirdSQL
InterBase
Oracle
Sybase
ODBC
Others?

Welcome to the Open Source Database Benchmark web site!

To date, there has been no easy way to benchmark the performance of a database system. The choices were

1. Hire a consulting group specializing in benchmarks
2. Purchase the rights to one of the important SPEC or TPC benchmarks, and staff a group to implement and run the tests
3. Review the results of published benchmarks and extrapolate them to your situation
4. Create your own benchmark suite

The cost of the first two options is well-justified in some cases. If one is about to spend ten million dollars on computer infrastructure (in addition to the cost of staffing, developing, and maintaining the application for which the project is being undertaken), spending one million dollars to confirm that you are about to make the right choice is very sensible.

Option 3 is a good choice if you are quite sure that the published benchmarks are relevant for your situation, and that you have the expertise to tune the systems as well as the vendors who prepared their systems for the benchmarks.

Note that many database vendors use licenses which prohibit the public disclosure of database performance information without their prior approval. Is this because of the difficulty of tuning their systems correctly, or fear that real-world performance will vary from their published benchmarks?

Option 4 is great if you have the time, resources, and expertise to create a benchmark test suite.

If you lack the time and resources, or if you lack confidence in your ability to construct a robust, reliable, repeatable benchmark, what can you do?

OSDB was created to provide a solid, peer-reviewedfoundation for people of sufficient technical skill to get a head start in running their own analysis of database and system performance, and to share the results of the labor (and joy!) in preparing this code.

Peer review is an important attribute of an open source project. The leader of the OSDB project is a software practitioner of varying skill in the arts of coding, databases, and benchmarks. By pooling his efforts with those of others, OSDB should become the robust, reliable, repeatable benchmark that it can be!

Sharing the results

In describing his invention, the "Franklin" stove, Franklin wrote:

"Gov'r. Thomas was so pleas'd with the construction of this stove, as described in it, that he offered to give me a patent for the sole vending of them for a term of years; but I declin'd it from a principle which has ever weighed with me on such occasions, viz., That, as we enjoy great advantages from the inventions of others, we should be glad of an opportunity to serve others by any invention of ours; and this we should do freely and generously."

From The Autobiography of Benjamin Franklin, Part VI atwww.wakeamerica.com

None of this work would have been possible without the nearly infinite number of inventions and ideas on which it is built. In that spirit, we hope to contribute in some small way to the global pool of intellectual wealth.

Headlines

• 30 Nov 2009
  New direction!
  As shared with the mailing lists earlier this month, the OSDB project has a new maintainer. My name is William Bartig, and I am excited to continue to move OSDB towards the future. Many thanks go to Andy for his work with OSDB for all of these years, and I certainly hope he is able to find time to check on us now and then.
  I have many plans for OSDB as we move forward, but for now, I will quote the first three goals that I expressed to the mailing lists:
  1. I will be updating the Oracle test to fix some problems it has, including hard-coded values and a problematic multi-user test.
  2. Port OSDB to an OO model in C++, which will make it much easier to develop tests. During the port, I will also work on the build system to make that a little easier to use and consistent between each type of test.
  3. After the first two goals are met, I plan to spread the word about OSDB through various OSS channels, which I hope will bring more interest (and more contributors) to the project.
  I welcome any kind of input on the project, and I look forward to working with everyone to bring an improved OSDB to more people who could benefit from it.
• 19 Oct 2004
  OSDB, the Open Source Database Benchmark, version 0.17 has been released!
  Featuring ports to DataBlitz and Oracle, as well as a preliminary Perl implementation, this is the best iteration of OSDB yet!
  This version of OSDB can
  • Skip tests that are inappropriate for the database engine, with the --restrict switch
  • Run under MPI, for true cluster and multiprocessor benchmarks
    New results have been posted in the expected/ directory, suggesting useful ways of invoking OSDB.
    Barring results to the contrary in the next couple of weeks, this will soon be the new "stable" version of OSDB.
• 20 May 2003
  Preview OSDB x0.15-1
  The osdb-unstable x0.15-1 kit, available from the SourceForge site, introduces "dbuser" and "dbpassword" options, allowing one to specify the database username and password.
• 11 May 2003
  Preview the next version of OSDB
  The osdb-unstable x0.15-0 kit, available from the SourceForge site, includes the ability to generate your own data to test databases from 4MB to 400GB. This kit lacks Informix functionality, and the -ui interface tests. As always, let us know of any problems.
• 13 March 2002
  Improved Support for MySQL and PostgreSQL Benchmarks
  The "Unlucky release" of OSDB, version 0.13, was quickly replaced with version 0.14 which introduces support for database product-specific options. Specifically, for MySQL, we now support the InnoDB and BDB engines via the
  --mysql={innodb | bdb}
  option. For you long-suffering OSDB PostgreSQL users, we offer
  --postgresql=no_hash_index
  to work around the hash index problems of OSDB with PostgreSQL V7.1 and 7.2. As always, let us know of any problems. May the source be with you!
• 20 February 2002
  OSDB version 0.12 releases with improved portability and support for MySQL 3.23, PostgreSQL 7.2
  OSDB version 0.12 features improved portability to most *nix systems, and a new build procedure that lets one specify which of several possible database installations should be installed.
• 15 February 2002
  OSDB joins the SourceForge DataBase Foundry!
• 30 September 2001
  OSDB version 0.11-- The silent release
  we forgot to tell anyone about it!
  OSDB version 0.11 features all new documentation, sample output, corrections to some of the SQL code, and a script that makes it possible to generate your own data sets (rather than downloading our sample data.
• 19 April 2001
  OSDB version 0.10 released with pretty good support for Informix®, MySQL®, and PostgreSQL®

This version has pretty good support for:

In addition, this version supports tests via the user interface programs (dbaccess, mysql, psql), but these are not as reliable as the embedded SQL or native API tests.