[Python-Dev] Benchmarks: Comparison between Python 2.7 and Python 3.6 performance (original) (raw)

Victor Stinner victor.stinner at gmail.com
Fri Nov 4 10:50:42 EDT 2016

The nice thing with benchmarks is that you can always take a subset of numbers to prove something :-) Another subset where Python 3.6 is faster:

"Python 3.6 is between 12% (1.14x) and 97% (32x) FASTER than Python 2.7 in the following benchmarks:" https://twitter.com/VictorStinner/status/794525289623719937

Victor

2016-11-04 13:53 GMT+01:00 Victor Stinner <victor.stinner at gmail.com>:

(Re-post without the two attached files of 100 KB each.)

Hi, You may know that I'm working on benchmarks. I regenerated all benchmark results of speed.python.org using performance 0.3.2 (benchmark suite). I started to analyze results. All results are available online on the website: https://speed.python.org/

To communicate on my work on benchmarks, I tweeted two pictures: "sympy benchmarks: Python 3.6 is between 8% and 48% faster than Python 2.7 #python #benchmark": https://twitter.com/VictorStinner/status/794289596683210760 "Python 3.6 is between 25% and 54% slower than Python 2.7 in the following benchmarks": https://twitter.com/VictorStinner/status/794305065708376069 Many people were disappointed that Python 3.6 can be up to 54% slower than Python 2.7. In fact, I know many reasons which explain that, but it's hard to summarize them in 140 characters ;-) For example, Python 3.6 is 54% slower than Python 2.7 on the benchmark pycryptoaes. This benchmark tests a pure Python implementation of the crypto cipher AES. You may know that CPython is slow for CPU intensive functions, especially on integer and floatting point numbers. "int" in Python 3 is now "long integers" by default, which is known to be a little bit slower than "short int" of Python 2. On a more realistic benchmark (see other benchmarks), the overhead of Python 3 "long int" is negligible. AES is a typical example stressing integers. For me, it's a dummy benchmark: it doesn't make sense to use Python for AES: modern CPUs have an hardware implemention which is super fast. Well, I didn't have time to analyze in depth individual benchmarks. If you want to help me, here is the source code of benchmarks: https://github.com/python/performance/blob/master/performance/benchmarks/ Raw results of Python 3.6 compared to Python 2.7: ------------------- $ python3 -m perf compareto 2016-11-0315-36-2.7-91f024fc9b3a.json.gz 2016-11-0315-38-3.6-c4319c0d0131.json.gz -G --min-speed=5 Slower (40): - pythonstartup: 7.74 ms +- 0.28 ms -> 26.9 ms +- 0.6 ms: 3.47x slower - pythonstartupnosite: 4.43 ms +- 0.08 ms -> 10.4 ms +- 0.4 ms: 2.36x slower - unpicklepurepython: 417 us +- 3 us -> 918 us +- 14 us: 2.20x slower - callmethod: 16.3 ms +- 0.2 ms -> 28.6 ms +- 0.8 ms: 1.76x slower - callmethodslots: 16.2 ms +- 0.4 ms -> 28.3 ms +- 0.7 ms: 1.75x slower - callmethodunknown: 18.4 ms +- 0.2 ms -> 30.8 ms +- 0.8 ms: 1.67x slower - cryptopyaes: 161 ms +- 2 ms -> 249 ms +- 2 ms: 1.54x slower - xmletreeparse: 201 ms +- 5 ms -> 298 ms +- 8 ms: 1.49x slower - loggingsimple: 26.4 us +- 0.3 us -> 38.4 us +- 0.7 us: 1.46x slower - loggingformat: 31.3 us +- 0.4 us -> 45.5 us +- 0.8 us: 1.45x slower - picklepurepython: 986 us +- 9 us -> 1.41 ms +- 0.03 ms: 1.43x slower - spectralnorm: 208 ms +- 2 ms -> 287 ms +- 2 ms: 1.38x slower - loggingsilent: 660 ns +- 7 ns -> 865 ns +- 31 ns: 1.31x slower - chaos: 240 ms +- 2 ms -> 314 ms +- 4 ms: 1.31x slower - go: 490 ms +- 2 ms -> 640 ms +- 26 ms: 1.31x slower - xmletreeiterparse: 178 ms +- 2 ms -> 230 ms +- 5 ms: 1.29x slower - sqlitesynth: 8.29 us +- 0.16 us -> 10.6 us +- 0.2 us: 1.28x slower - xmletreeprocess: 210 ms +- 6 ms -> 268 ms +- 14 ms: 1.28x slower - djangotemplate: 387 ms +- 4 ms -> 484 ms +- 5 ms: 1.25x slower - fannkuch: 830 ms +- 32 ms -> 1.04 sec +- 0.03 sec: 1.25x slower - hexiom: 20.2 ms +- 0.1 ms -> 24.7 ms +- 0.2 ms: 1.22x slower - chameleon: 26.1 ms +- 0.2 ms -> 31.9 ms +- 0.4 ms: 1.22x slower - regexcompile: 395 ms +- 2 ms -> 482 ms +- 6 ms: 1.22x slower - jsondumps: 25.8 ms +- 0.2 ms -> 31.0 ms +- 0.5 ms: 1.20x slower - nqueens: 229 ms +- 2 ms -> 274 ms +- 2 ms: 1.20x slower - genshitext: 81.9 ms +- 0.6 ms -> 97.8 ms +- 1.1 ms: 1.19x slower - raytrace: 1.17 sec +- 0.03 sec -> 1.39 sec +- 0.03 sec: 1.19x slower - scimarkmontecarlo: 240 ms +- 7 ms -> 282 ms +- 10 ms: 1.17x slower - scimarksor: 441 ms +- 8 ms -> 517 ms +- 12 ms: 1.17x slower - deltablue: 17.4 ms +- 0.1 ms -> 20.1 ms +- 0.6 ms: 1.16x slower - sqlalchemydeclarative: 310 ms +- 3 ms -> 354 ms +- 6 ms: 1.14x slower - callsimple: 12.2 ms +- 0.2 ms -> 13.9 ms +- 0.2 ms: 1.14x slower - scimarkfft: 613 ms +- 19 ms -> 694 ms +- 23 ms: 1.13x slower - meteorcontest: 191 ms +- 1 ms -> 215 ms +- 2 ms: 1.13x slower - pathlib: 46.9 ms +- 0.4 ms -> 52.6 ms +- 0.9 ms: 1.12x slower - richards: 181 ms +- 1 ms -> 201 ms +- 6 ms: 1.11x slower - genshixml: 191 ms +- 2 ms -> 209 ms +- 2 ms: 1.10x slower - float: 290 ms +- 5 ms -> 310 ms +- 7 ms: 1.07x slower - scimarksparsematmult: 8.19 ms +- 0.22 ms -> 8.74 ms +- 0.15 ms: 1.07x slower - xmletreegenerate: 302 ms +- 3 ms -> 320 ms +- 8 ms: 1.06x slower Faster (15): - telco: 707 ms +- 22 ms -> 22.1 ms +- 0.4 ms: 32.04x faster - unpicklelist: 15.0 us +- 0.3 us -> 7.86 us +- 0.16 us: 1.90x faster - picklelist: 14.7 us +- 0.2 us -> 9.12 us +- 0.38 us: 1.61x faster - jsonloads: 98.7 us +- 2.3 us -> 62.3 us +- 0.7 us: 1.58x faster - pickle: 40.4 us +- 0.6 us -> 27.1 us +- 0.5 us: 1.49x faster - sympysum: 361 ms +- 10 ms -> 244 ms +- 7 ms: 1.48x faster - sympyexpand: 1.68 sec +- 0.02 sec -> 1.15 sec +- 0.03 sec: 1.47x faster - regexv8: 62.0 ms +- 0.5 ms -> 47.2 ms +- 0.6 ms: 1.31x faster - sympystr: 699 ms +- 22 ms -> 537 ms +- 15 ms: 1.30x faster - regexeffbot: 6.67 ms +- 0.04 ms -> 5.23 ms +- 0.05 ms: 1.28x faster - mako: 61.5 ms +- 0.7 ms -> 49.7 ms +- 2.5 ms: 1.24x faster - html5lib: 298 ms +- 7 ms -> 261 ms +- 6 ms: 1.14x faster - sympyintegrate: 55.9 ms +- 0.3 ms -> 51.8 ms +- 1.0 ms: 1.08x faster - pickledict: 69.4 us +- 0.9 us -> 65.2 us +- 3.2 us: 1.06x faster - scimarklu: 551 ms +- 26 ms -> 523 ms +- 18 ms: 1.05x faster Benchmark hidden because not significant (8): 2to3, dulwichlog, nbody, pidigits, regexdna, tornadohttp, unpacksequence, unpickle Ignored benchmarks (3) of 2016-11-0315-36-2.7-91f024fc9b3a.json: hgstartup, pyflate, spambayes ------------------- Please ignore callmethod, callmethodslots, callmethodunknown benchmarks: it seems like I had an issue on the benchmark server. I was unable to reproduce he 70% slowdown on my laptop. JSON files if you want to analyze them yourself: http://www.haypocalc.com/tmp/2016-11-0315-36-2.7-91f024fc9b3a.json.gz http://www.haypocalc.com/tmp/2016-11-0315-38-3.6-c4319c0d0131.json.gz I hope that my work on benchmarks will motive some developers to look closer at Python 3 performance to find interesting optimizations ;-) Victor

More information about the Python-Dev mailing list