Lehigh University Benchmark (LUBM) (original) (raw)

Overview

The Lehigh University Benchmark is developed to facilitate the evaluation of Semantic Web repositories in a standard and systematic way. The benchmark is intended to evaluate the performance of those repositories with respect to extensional queries over a large data set that commits to a single realistic ontology. It consists of a university domain ontology, customizable and repeatable synthetic data, a set of test queries, and several performance metrics.

References:
For an in-depth description of the benchmark and some evaluations done with its OWL version, refer to:

• [4] "LUBM: A Benchmark for OWL Knowledge Base Systems"
• [3] "An Evaluation of Knowledge Base Systems for Large OWL Datasets"

For some evaluations with the benchmark's DAML+OIL version, refer to: * [2] "Choosing the Best Knowledge Base System for Large Semantic Web Applications" * [1] "Benchmarking DAML+OIL Repositories"

Components

• Ontology: The benchmark ontology is named Univ-Bench: OWL Version
• Data Generator(UBA): This tool generates syntetic OWL or DAML+OIL data over the Univ-Bench ontology in the unit of a university. These data are repeatable and customizable, by allowing user to specify seed for random number generation, the number of universities, and the starting index of the universities.
  • What do the data look like?
• Test Queries: The benchmark currently provides 14 test queries. The file contains all queries in SPARQL 1.0 syntax, separated by blank lines and identify comments. Examples of using other languages such as RQL can be found in the download of the test module below. Here is a visualization of the queries in JPG and SVG format (thanks to Mike Dean of BBN).
• Tester (UBT): The test module. It carries out both data loading test and query test, with configurable test plans.

Downloads

This project is currently hosted at SemWebCentral. The software suite contains the data generator (UBA) and the tester (UBT).

• Data Generator(UBA):
  • UBA1.7: Minor changes including package name. Using options -index 0 -seed 0 can produce the same data sets as used in papers [4, 3].
  • UBA1.6: Supports both OWL and DAML+OIL. Also there were some changes in the generated data from the previous version.
  • UBA1.4: Supports DAML+OIL only. Using options -index 0 -seed 0 can produce the same data sets as used in papers [2, 1].
• Tester (UBT):
  • UBT1.1
• Others: (Courtesy of Mikael Högqvist)
  • Linux file path fix: Makes UBA1.7 work with Linux paths.
  • Ant build file: An Ant build file for UBA.

Documentation

• Repository API: This document describes the interface between the benchmark tester and a target repository, including those that are required to be implemented in the repository (i.e. QueryResult, Repository and RepositoryFactory).

Others

• Reference Query Answers: Reference query answers for LUBM(1,0). (The dataset is generated by UBA1.7.)
• A sample wrapper for Sesame 1.0.

Contact

Jeff Heflin