A family of real-time Java benchmarks (original) (raw)

2011, Concurrency and Computation: Practice and Experience

Java is becoming a viable platform for real-time computing. There are production and research real-time Java VMs, as well as applications in both military and civil sector. Technological advances and increased adoption of Real-time Java contrast significantly with the lack of benchmarks. Existing benchmarks are either synthetic micro-benchmarks, or proprietary, making it difficult to independently verify and repeat reported results. This paper presents the CDx benchmark, a family of open source implementations of the same application that target different real-time virtual machines. CDx is, at its core, a real-time benchmark with a single periodic task, which implements an idealized aircraft collision detection algorithm. The benchmark can be configured to use different sets of real-time features and comes with a number of workloads. It can be run on standard Java virtual machines, on real-time and Safety Critical Java virtual machine, and a C version is provided to compare with native performance. be used to evaluate the quality of real-time virtual machines, they are not representative of real-time workloads. They aim to generate the highest sustainable load and they measure the mean performance under this load, neither of which suits real-time systems. Real-time systems are designed such that deadlines are not missed. The ability to meet deadlines depends on the worst-case computation times of periodically scheduled tasks, which are not captured by mean performance metrics. Moreover, programming styles for real-time systems are very different from non-realtime throughput targeted systems. The usefulness of benchmarks for performance evaluation is that the benchmarks, being realistic models of real applications, put the system of interest under a workload similar to those real applications. They allow us to capture and evaluate performance characteristics caused by many aspects of program execution, some of which we may not be aware of or be able to predict. For realtime Java, we thus need benchmarks that actually model real-time systems, have deadlines, use RTSJ, run on real-time OS kernels, use high precision timers, and measure workloads configured to never miss a deadline. Unfortunately, it is notoriously difficult to find real-time applications in the wild. Most real-time systems are proprietary and are tied to some hardware/OS platform. Real-time Java being a relatively young technology does not help.