Performance Comparison of Java/.NET Runtimes (Oct 2004) (original) (raw)
last-updated: November 20 21:55, 2005 JST
Back to:
Performance comparison pages:
- Oct 2004: SPEC JVM98, SciMark 2.0, Linpack benchmark and Eratosthenes Sieve on Pentium 4 and Opteron.
- Jan 2002: SPEC JVM98, SciMark 2.0, Linpack benchmark and Eratosthenes Sieve on Pentium 4.
- Feb 2001: SPEC JVM98, SciMark 2.0, Linpack benchmark and Eratosthenes Sieve on Pentium III, K6-2 and Crusoe.
- Sep 2000: SPEC JVM98, SciMark 2.0, Linpack benchmark and Eratosthenes Sieve on K6-2.
- Mar 1999: Eratosthenes Sieve, Linpack benchmark and CaffeineMark 3.0 on Pentium.
News
November 20, 2005
Linpack.cs and Linpack.java were updated and show correct scores with a large matrix size about 1,300.
Thanks Dr. Tobias Kind.
December 17, 2004
Scores of Mono 1.1.3 were added.
December 12, 2004
Linpack.cs and Linpack.java were added.
October 29, 2004
JRockit was measured and the scores were added.
October 27, 2004
Results of Linpack benchmark in C added.
Numbers of Visual C++ / SciMark 2 updated.
October 25, 2004
Results of SciMark 2 in C# and C added.
October 19, 2004
Mono was measured and the scores were added.
October 14, 2004
This page was initially provided.
This page shows results of performance comparisons among Java runtime systems on Linux and .NET runtime system on Windows.
Applied benchmarks
- SPEC JVM98 (in Java)
- SciMark 2.0 (in Java, C# and C)
- Linpack benchmark (in Java, C# and C)
- Eratosthenes Sieve (in Java and C#)
Evaluated Java runtime systems
- Sun JDK 1.5.0
- HotSpot Client VM
- HotSpot Server VM
- Interpreter
- IBM JDK 1.4.2 GA (build cxia321420-20040626)
- JITC 4.02 included.
- IBM JDK 1.3.1 SR7P (build cxia32131ifx-20040727)
- JITC 4.02 included.
- BEA WebLogic JRockit 8.1 (1.4.2_04 JVM) (build ari-31788-20040616-1032-linux-ia32)
- This runtime accepts -server and -client options. The default setting of java command (for all benchmarks except SPEC JVM98) is -server and appletviewer (for SPEC JVM98) uses -client.
- Blackdown J2SDK 1.3.1_02b FCS
- Jikes RVM 2.3.3
(released at July 28, 2004)- Configured with the command `jconfigure production'.
- Kaffe 1.1.4
- "Just-in-time v3" included.
- GCJ 3.4.2 and 3.3.3
- GCJ is the GNU Compiler for the Java Programming Language.
- 3.4.2 for Pentium 4 and 3.3.3 for Opteron.
Evaluated .NET runtime systems
- Microsoft .NET Framework 1.1(English,Japanese)
- Version 1.1.4322 SP1.
- For C# version of Linpack benchmark and Eratosthenes Sieve.
- Only this runtime ran on Windows XP and others were on Linux.
- Mono 1.0.2 and 1.1.3
- 1.0.2 is a stable version and 1.1.3 is a development version.
Evaluated C compilers
- gcc: C front end of GCC 3.4.2
- icc: Intel(R) C++ compiler 8.1(product page)
- Build 20040921Z Package ID: l_cc_pc_8.1.022
- Microsoft Visual C++ .NET (Visual Studio.NET 2003)
- Microsoft(R) 32-bit C/C++ Optimizing Compiler Version 13.10.3077 for 80x86
Computers
- Pentium 4 PC
- Intel 3.2 GHz Pentium 4 (Northwood)
* Single processor
* Hyper-Threading disabled - Intel 875P chipset
- Linux 2.6.8
- Windows XP SP2 only for .NET Framework and Visual C++
- Intel 3.2 GHz Pentium 4 (Northwood)
- Opteron PC
- AMD 2.2 GHz Opteron (model 248)
* Dual processors - AMD 8131/8111 chipset
- Linux 2.6.8
- AMD 2.2 GHz Opteron (model 248)
All results are also available in Microsoft Excel (2003) book format: `<JIT-eval-20041217.xls>'.
SPEC JVM98 (in Java)
SPEC JVM98(http://www.spec.org/osg/jvm98/)is a benchmark suite consists of real Java application programs:
- Modified Lampel-Ziv method (_201_compress),
- Java Expert Shell System based on NASA's CLIPS expert shell system (_202_jess),
- Database functions performed on memory resident database (_209_db),
- Java compiler from the JDK 1.0.2 (_213_javac),
- MPEG Audio Layer-3 decoder (_222_mpegaudio),
- Ray-tracer where two threads work (_227_mtrt),
- and Java parser generator which is now called JavaCC (_228_jack).
SPEC JVM98 shows scores for each benchmark program and the geometric mean of them. Higher number is better.
Execution time of each benchmark program is rather short on a today's computer. For example, with IBM JDK 1.4.2, _202_jess, _222_mpegaudio and _228_jack finish only in 2 seconds and _227_mtrt takes only 0.83 second. SPEC JVM98 seems not to be able to evaluate advanced optimizing techniques for server-side applications like profile-guided optimizations.
Fairness of results
The results of Jikes RVM, Kaffe and GCJ are not reportable to SPEC because these runtime systems read benchmark programs and data from a local filesystem. Reportable results have to be produced by the benchmark which run as a Java applet. Jikes RVM and GCJ cannot run a Java applet and Kaffe does not work correctly with the applet version of SPEC JVM98.
_201_compress and _213_jack benchmarks are very sensitive to where those programs and data for them are read from. Thus the results of these two benchmarks cannot be compared fairly among all runtime systems.
Conditions
Benchmark programs and a web server (Apache httpd) ran on the same single computer and those programs and data were loaded from the web server (except Jikes RVM, Kaffe and GCJ).
Initial and maximum heap sizes remain as those default values. Note that one exception is JDK 1.5.0 for x86 on Opteron, whose default values are not enough to run SPEC JVM98 and initial and maximum heap sizes are set as 128 MB.
TYA 1.8 cannot run _200_check, _228_jack and _213_javac and then we use TYA 1.7v3.
Results
Pentium 4
Runtime system _227_ mtrt _202_ jess _201_ compress _209_ db _222_ mpegaudio _228_ jack _213_ javac Geometric Mean IBM JDK 1.4.2 557 256 324 74.1 650 247 165 265 IBM JDK 1.3.1 509 218 332 73.2 527 235 140 241 Sun JDK 1.5.0 Server VM 496 331 300 46.3 533 222 91.9 220 Sun JDK 1.4.2 Server VM 504 332 294 45.5 513 199 98.8 217 JRockit (-server) 265 289 282 73.7 485 204 123 212 JRockit (-client) 193 215 260 70.2 476 203 123 190 Sun JDK 1.5.0 Client VM 325 221 204 43.4 251 192 96.1 162 Sun JDK 1.4.2 Client VM 318 186 199 43.6 249 181 90.9 155 GCJ (-O2) 122 87.0 187 29.0 375 N/A 56.1 103 Jikes RVM 109 124 182 38.0 195 94.2 52.7 98.6 GCJ (-O0) 108 81.9 120 28.7 108 N/A 54.1 75.0 OpenJIT 82.1 60.7 177 16.1 123 52.7 42.5 62.9 shuJIT 45.8 53.2 117 17.2 114 51.0 37.2 52.2 Kaffe 32.0 21.3 191 24.8 101 32.9 21.3 41.9 TYA 33.9 46.1 83.4 15.1 67.7 43.5 34.4 41.1 sunwjit 15.5 23.8 145 16.3 86.6 43.8 33.7 37.8 interpreter 19.6 21.2 13.0 12.4 14.7 28.4 17.2 17.4
Opteron (AMD64 binary)
Runtime system _227_ mtrt _202_ jess _201_ compress _209_ db _222_ mpegaudio _228_ jack _213_ javac Geometric Mean Sun JDK 1.5.0 Server VM 650 270 354 46.6 538 351 114 251 GCJ (-O2) 207 108 260 35.4 429 N/A 77.1 138 GCJ (-O0) 133 88.8 99.8 34.1 75.7 N/A 70.1 77.3 interpreter 27.0 18.3 10.7 10.2 12.4 30.2 16.4 16.5
Opteron (x86 binary)
Runtime system _227_ mtrt _202_ jess _201_ compress _209_ db _222_ mpegaudio _228_ jack _213_ javac Geometric Mean JRockit (-server) 450 286 307 62.5 441 316 161 247 Sun JDK 1.5.0 Server VM 611 308 265 50.0 481 344 106 238 JRockit (-client) 236 256 264 64.2 453 298 161 217 Sun JDK 1.5.0 Client VM 467 236 228 46.2 262 294 97.4 189 GCJ (-O2) 133 99.6 187 35.1 311 N/A 76.1 113 GCJ (-O0) 108 86.3 107 33.9 75.9 N/A 69.9 75.0 interpreter 26.6 18.4 11.1 11.0 12.1 31.5 16.7 16.8
SciMark 2.0 (in Java, C# and C)
SciMark 2.0(http://math.nist.gov/scimark2/)is a benchmark suite consists of numerical kernels. C version is also available at the NIST site. C# version is available at a web page at Cornell University(http://rotor.cs.cornell.edu/SciMark/).
SciMark 2.0 shows results in MFlops (mega floating-point operations per second) and the composite score, which is the arithmetic mean of them. Higher number is better. Therefore, SOR showing relatively higher value affects the composite score much. Oppositely, FFT and Monte Carlo are not counted very much.
Fairness of results
C version is advantageous compared with Java and C# versions because Java and C# versions use a "synchronized" version of the random number generator, which makes the Monte Carlo Test much slower.
Conditions
We specified the `-large' option to run the benchmarks with the large data set.
We tried "-O2", "-O3" and "-O3 -funroll-loops" options with gcc and "-O2" yielded the best numbers. Similarly, "-O2", "-O3" and "-O3 -xN" tried with icc and "-O2" was the best.
Pentium 4
Runtime system FFT (1048576) SOR (1000x1000) Monte Carlo Sparse matmult (N=100000, nz=1000000) LU (1000x1000) Composite Score C/Visual C++ 27.7 761 90.9 384 384 329 Java/Sun JDK 1.4.2 Server VM 38.1 742 115 356 366 324 Java/Sun JDK 1.5.0 Server VM 34.4 741 94.3 347 366 316 C/icc (-O2) 30.9 641 85.5 391 380 306 C/gcc (-O2) 31.4 451 84.2 385 378 266 Java/IBM JDK 1.4.2 32.9 451 84.0 315 370 250 Java/JRockit (-server) 35.2 430 112 307 358 249 Java/IBM JDK 1.3.1 34.7 449 62.1 320 367 247 Java/JRockit (-client) 37.0 430 113 290 359 246 C#/.NET (Release) 25.9 454 46.9 323 348 240 Java/GCJ (-O2 with .java) 31.9 409 14.6 281 353 218 Java/GCJ (-O2 with .class) 31.9 409 14.4 238 338 206 Java/Sun JDK 1.5.0 Client VM 34.3 413 54.1 164 327 199 Java/Jikes RVM 24.2 412 7.12 261 279 197 Java/Sun JDK 1.4.2 Client VM 34.5 414 38.5 165 325 195 C#/Mono 1.0.2 (JIT) 34.4 355 2.12 178 245 163 C#/Mono 1.1.3 (JIT) 31.8 375 5.24 180 219 162 Java/Kaffe 29.2 226 10.6 95.3 216 115 Java/GCJ (-O0 with .java) 27.7 193 13.1 106 153 98.6 Java/OpenJIT 28.6 124 8.5 123 55.7 68.0 Java/GCJ (-O0 with .class) 23.1 132 11.7 66.1 83.1 63.2 Java/sunwjit 31.8 148 5.73 59.6 18.6 52.7 Java/shuJIT 17.1 100 7.52 58.6 65.4 49.7 Java/TYA 14.2 64.2 5.93 50.3 71.5 41.2 Java/interpreter 6.63 26.9 4.88 19.8 41.9 20.0 C#/Mono 1.1.3 (interpreter) 4.11 18.4 0.83 9.98 7.46 8.16 C#/Mono 1.0.2 (interpreter) 4.22 16.4 0.83 8.24 8.12 7.56
Opteron (AMD64 binary)
Runtime system FFT (1048576) SOR (1000x1000) Monte Carlo Sparse matmult (N=100000, nz=1000000) LU (1000x1000) Composite Score Sun JDK 1.5.0 Server VM 36.9 669 73.3 249 299 265 GCJ (-O2 with .class) 38.2 481 37.0 235 288 216 GCJ (-O2 with .java) 38.1 481 32.4 229 285 213 GCJ (-O0 with .java) 24.4 146 21.6 80.2 109 76.3 GCJ (-O0 with .class) 21.1 120 17.0 70.5 62.5 58.2 interpreter 5.84 24.5 3.73 15.1 20.2 13.9
Opteron (x86 binary)
Runtime system FFT (1048576) SOR (1000x1000) Monte Carlo Sparse matmult (N=100000, nz=1000000) LU (1000x1000) Composite Score Sun JDK 1.5.0 Server VM 37.8 659 64.4 234 300 259 JRockit (-server) 41.6 463 121 258 301 237 JRockit (-client) 41.5 468 121 254 289 235 GCJ (-O2 with .class) 37.7 482 29.8 228 293 214 GCJ (-O2 with .java) 38.1 476 28.9 217 302 212 Sun JDK 1.5.0 Client VM 33.9 479 41.2 186 204 189 GCJ (-O0 with .java) 23.4 147 20.9 80.9 99.9 74.5 GCJ (-O0 with .class) 19.9 118 16.4 66.4 65.6 57.3 interpreter 5.52 21.7 3.62 13.7 19.5 12.8
Linpack benchmark (in Java, C# and C)
SeeFAQ on the LINPACK Benchmark. The results are in MFlops (mega floating-point operations per second), so higher is better.
We used a modified version of the Java version(http://www.netlib.org/benchmark/linpackjava/), the C version(http://www.netlib.org/benchmark/linpackc)and ported the Java version to C#:
- <Linpack.java>
- <Linpack.cs>
All these versions of Linpack benchmark are not well optimized (ex. not blocked) and do not exploit potential performance of runtimes/platforms much because the purpose of this benchmarking is not such exploitation but comparison of runtimes/platforms.
Pentium 4
| 1000 x 1000 | 500 x 500 | | | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | |
|
| | | MFlopssecond C/icc (-O3 -xN)402.5191.66 C/Visual C++ (/Ox /G7 /arch:SSE2)399.9201.67 C/gcc (-O3 -funroll-loops)393.6671.70 C#/.NET (Release)382.0991.75 Java/Sun JDK 1.5.0 Server VM379.9241.76 Java/Sun JDK 1.4.2 Server VM379.2781.763 Java/IBM JDK 1.4.2374.3931.786 Java/IBM JDK 1.3.1360.8561.853 Java/GCJ (-O2 with .java)357.0031.873 Java/GCJ (-O2 with .class)348.8091.917 Java/Sun JDK 1.5.0 Client VM339.9421.967 Java/Sun JDK 1.4.2 Client VM339.081.972 Java/JRockit (-server)318.2612.101 Java/JRockit (-client)318.112.102 C#/Mono 1.1.3 (JIT)313.1132.136 Java/Jikes RVM309.9752.207 C#/Mono 1.0.2 (JIT)230.6512.899 Java/Kaffe211.0023.169 C#/.NET (Debug)195.4093.422 Java/GCJ (-O0 with .java)151.1794.423 Java/OpenJIT130.3445.13 Java/GCJ (-O0 with .class)92.3067.244 Java/shuJIT79.8038.379 Java/TYA71.8159.311 Java/interpreter26.00625.712 Java/sunwjit15.83342.233 C#/Mono 1.1.3 (interpreter)9.4770.611 C#/Mono 1.0.2 (interpreter)6.591101.452 | MFlopssecond C/Visual C++ (/Ox /G7 /arch:SSE2)448.3070.19 C/icc (-O3 -xN)433.9740.19 C/gcc (-O3 -funroll-loops)413.4550.20 Java/GCJ (-O2 with .java)359.80.233 Java/JRockit (-server)349.3060.24 Java/JRockit (-client)349.3060.24 Java/GCJ (-O2 with .class)349.3060.24 Java/Sun JDK 1.5.0 Server VM319.9750.262 Java/IBM JDK 1.4.2317.5510.264 Java/Sun JDK 1.5.0 Client VM317.5510.264 Java/IBM JDK 1.3.1312.8110.268 Java/Sun JDK 1.4.2 Client VM312.8110.268 C#/Mono 1.1.3 (interpreter)301.9980.278 Java/Sun JDK 1.4.2 Server VM297.2810.282 C#/.NET (Release)282.3860.297 C#/Mono 1.0.2 (JIT)222.4290.377 Java/Kaffe198.1880.423 Java/Jikes RVM192.2780.436 C#/.NET (Debug)162.5880.516 Java/GCJ (-O0 with .java)151.5970.553 Java/OpenJIT129.9740.645 Java/GCJ (-O0 with .class)90.9260.922 Java/shuJIT76.771.092 Java/TYA70.5671.188 Java/interpreter25.6533.268 Java/sunwjit15.4535.425 C#/Mono 1.1.3 (interpreter)8.6769.663 C#/Mono 1.0.2 (interpreter)6.56812.764 | |
Opteron
1000 x 1000 500 x 500 AMD binary x86 binary AMD binary x86 binary MFlops second MFlops second MFlops second MFlops second Sun JDK 1.5.0 Server VM 304.216 2.198 297.582 2.247 311.648 0.269 318.758 0.263 JRockit (-server) 273.371 2.446 266.985 0.314 JRockit (-client) 273.26 2.447 268.697 0.312 GCJ (-O2 with .java) 276.881 2.415 240.441 2.781 289.08 0.29 250.998 0.334 GCJ (-O2 with .class) 275.171 2.43 220.682 3.03 287.1 0.292 225.966 0.371 Sun JDK 1.5.0 Client VM 217.665 3.072 222.961 0.376 GCJ (-O0 with .java) 103.959 6.432 93.625 7.142 103.754 0.808 94.089 0.891 GCJ (-O0 with .class) 64.812 10.317 64.512 10.365 65.291 1.284 64.736 1.295 interpreter 14.365 46.548 13.954 47.918 14.209 5.9 13.784 6.082
Eratosthenes Sieve (in Java and C#)
This benchmark is the Sieve.javain the TYA archive. These values are number of operations per 10 seconds, so higher is better.
Pentium 4
IBM JDK 1.4.2 13327 IBM JDK 1.3.1 13308 JRockit (-server) 13197 GCJ (-O2 with .java) 13141 JRockit (-client) 12991 C#/.NET (Release) 12735 GCJ (-O2 with .class) 12057 Sun JDK 1.4.2 Server VM 11992 Sun JDK 1.5.0 Server VM 11174 Jikes RVM 9555 C#/Mono 1.1.3 (JIT) 9420 Kaffe 9024 C#/.NET (Debug) 8217 C#/Mono 1.0.2 (JIT) 8047 Sun JDK 1.4.2 Client VM 7546 Sun JDK 1.5.0 Client VM 7275 OpenJIT 6587 sunwjit 6269 GCJ (-O0 with .java) 5488 shuJIT 5005 TYA 4756 GCJ (-O0 with .class) 4350 interpreter 515 C#/Mono 1.0.2 (interpreter) 343 C#/Mono 1.1.3 (interpreter) 337
Opteron
AMD64 binary x86 binary GCJ (-O2 with .java)13522 GCJ (-O2 with .class)11934 Sun JDK 1.5.0 Server VM10736 GCJ (-O0 with .java)3471 GCJ (-O0 with .class)2830 interpreter537 JRockit (-server)13460 JRockit (-client)13412 Sun JDK 1.5.0 Server VM12906 GCJ (-O2 with .java)12771 GCJ (-O2 with .class)10653 Sun JDK 1.5.0 Client VM6631 GCJ (-O0 with .java)3934 GCJ (-O0 with .class)3478 interpreter552
Related Resources
The Java Performance Report
http://www.javalobby.org/members/jpr/
Java Grande Forum
Java Numerics (at NIST)
http://math.nist.gov/javanumerics/
validgh Java numerics page
Java in Science and Engineering list, Java Grande Forum list
http://mailer.csit.fsu.edu/mailman/listinfo
Copyright (C) 2004 Kazuyuki Shudo