Announcing the Renaissance benchmark suite (original) (raw)

Francois Farquet francois.farquet at oracle.com
Mon May 13 14:10:58 UTC 2019

Le 13.05.19 à 11:54, Florian Weimer a écrit :

* Alan Bateman:

On 09/05/2019 19:04, Francois Farquet wrote: Hi Alan,

The suite has been developed and tested under JDK8 and full support of JDK11 is planned for the 1.0.0 release. Regarding the latest JDK and ea versions, we’ll do our best to test it regularly but if the Java Product Group at Oracle could regularly check its compatibility and performance, that would be great. From what I can tell, none of the apache-spark benchmarks will run with with any recent feature releases. They all fall over with HADOOP-14586 in code that is parsing the value of "java.version" property. The database benches seems to have issues too, seems to be code in one of Chronicle libraries trying to hack java.lang.reflect.AccessibleObject privates. Are you, or others, working with these projects to get fixes or versions that run on more recent releases, maybe as part of the effort to get it to run on JDK 11 builds? This leads to the question to what extent these higher-level run-time environments are part of the benchmark. Is it expected they remain unchanged (except for such bug fixes)? Or can they be enhanced to use current and new JVM features that help with their implementation?

Each benchmark should be made as compatible as possible with all reasonable JDK versions.

However, that doesn't mean there shouldn't be benchmarks that are compatible only with some late JDK version if there is a good reason for that, like the use of recent features that are representative of what a recent workload looks like. So, considering Renaissance as an evolving benchmark suite with benchmarks being added or removed from a release to another, we can imagine having some JDK version restriction per benchmark if we face that need.

Regarding enhancing current benchmarks, I would advise against anything different than bug fixes. Major changes should be made as a distinct benchmark to avoid confusion.

Thanks,

Francois

Thanks, Florian

More information about the jdk-dev mailing list