[Python-Dev] Python 3.3 vs. Python 2.7 benchmark results (again, but this time more solid numbers) (original) (raw)

Brett Cannon brett at python.org
Fri Oct 26 21:14:08 CEST 2012

I re-ran the unladen benchmarks on my work machine and w/o the -b option flipped on (i.e. more thorough benchmark numbers). I figured I would share them now instead of after my PyCon Argentina talk in case people decide to dig into the results now, find a pathological problem in CPython, and then fix it before I give my presentation (if you have trouble running a benchmark or it isn't available in the repo because it's one I hacked together, just ask and I can help you run the benchmark if you want to try to speed things up). I have colour-coded benchmarks based on whether it is faster or slower in Python 3.3 (sorry for those of you who hate HTML email).

But the tl;dr message is that Python 3.3 looks good compared to Python 2.7 (the median benchmark score is 5% slower).

Worst benchmark is nosite_startup, best is telco. The benchmarks people might want to analyze (i.e. more than 20% slower in Python 3.3) are mako_v2, threaded_count, normal_startup, iterative_count, pathlib, formatted_logging, and simple_logging.

###########################################

Report on Linux 3.2.5-gg987 #1 SMP Fri Sep 14 02:36:36 PDT 2012 x86_64 x86_64 Total CPU cores: 12

2to3

9.320000 -> 8.980000: 1.04x faster

call_method

Min: 0.417756 -> 0.355247: 1.18x faster Avg: 0.419688 -> 0.356382: 1.18x faster Significant (t=92.85) Stddev: 0.00604 -> 0.00577: 1.0479x smaller Timeline: b'http://tinyurl.com/8soruze'

call_method_slots

Min: 0.417611 -> 0.358451: 1.17x faster Avg: 0.420761 -> 0.359676: 1.17x faster Significant (t=88.70) Stddev: 0.00605 -> 0.00588: 1.0291x smaller Timeline: b'http://tinyurl.com/8uu8234'

call_method_unknown

Min: 0.459057 -> 0.359327: 1.28x faster Avg: 0.462929 -> 0.360410: 1.28x faster Significant (t=137.99) Stddev: 0.00698 -> 0.00583: 1.1969x smaller Timeline: b'http://tinyurl.com/9mo7h24'

call_simple

Min: 0.341689 -> 0.265289: 1.29x faster Avg: 0.343003 -> 0.266503: 1.29x faster Significant (t=124.20) Stddev: 0.00555 -> 0.00511: 1.0859x smaller Timeline: b'http://tinyurl.com/9pnn7q4'

chameleon

Min: 0.072232 -> 0.062713: 1.15x faster Avg: 0.074588 -> 0.064261: 1.16x faster Significant (t=33.74) Stddev: 0.00284 -> 0.00245: 1.1599x smaller Timeline: b'http://tinyurl.com/8my8afl'

chaos

Min: 0.313727 -> 0.367015: 1.17x slower Avg: 0.317568 -> 0.371473: 1.17x slower Significant (t=-26.72) Stddev: 0.00962 -> 0.01053: 1.0942x larger Timeline: b'http://tinyurl.com/9y2u6kh'

django

Min: 0.798331 -> 0.855461: 1.07x slower Avg: 0.801109 -> 0.860996: 1.07x slower Significant (t=-87.43) Stddev: 0.00336 -> 0.00348: 1.0356x larger Timeline: b'http://tinyurl.com/9sf95pq'

fannkuch

Min: 1.364705 -> 1.327680: 1.03x faster Avg: 1.380412 -> 1.337467: 1.03x faster Significant (t=10.48) Stddev: 0.02056 -> 0.02040: 1.0077x smaller Timeline: b'http://tinyurl.com/9r2vq6g'

fastpickle

Min: 0.763479 -> 0.805715: 1.06x slower Avg: 0.770036 -> 0.810855: 1.05x slower Significant (t=-12.73) Stddev: 0.01618 -> 0.01589: 1.0180x smaller Timeline: b'http://tinyurl.com/9rvqo4s'

fastunpickle

Min: 0.588694 -> 0.663616: 1.13x slower Avg: 0.596622 -> 0.672418: 1.13x slower Significant (t=-23.22) Stddev: 0.01503 -> 0.01752: 1.1656x larger Timeline: b'http://tinyurl.com/9eggn34'

float

Min: 0.363234 -> 0.344408: 1.05x faster Avg: 0.376159 -> 0.354165: 1.06x faster Significant (t=8.76) Stddev: 0.01282 -> 0.01227: 1.0455x smaller Timeline: b'http://tinyurl.com/8d6rcb8'

formatted_logging

Min: 0.330988 -> 0.400309: 1.21x slower Avg: 0.335522 -> 0.408920: 1.22x slower Significant (t=-33.48) Stddev: 0.00989 -> 0.01194: 1.2076x larger Timeline: b'http://tinyurl.com/9ll7dqk'

genshi

Min: 0.229140 -> 0.251766: 1.10x slower Avg: 0.232124 -> 0.257252: 1.11x slower Significant (t=-40.24) Stddev: 0.00516 -> 0.00564: 1.0925x larger Timeline: b'http://tinyurl.com/9dpuuaw'

go

Min: 0.632778 -> 0.710382: 1.12x slower Avg: 0.636143 -> 0.716748: 1.13x slower Significant (t=-37.61) Stddev: 0.00186 -> 0.01504: 8.0815x larger Timeline: b'http://tinyurl.com/8s7vw74'

hexiom2

Min: 150.982155 -> 154.702444: 1.02x slower Avg: 151.194622 -> 154.780953: 1.02x slower Significant (t=-15.83) Stddev: 0.30047 -> 0.11103: 2.7063x smaller Timeline: b'http://tinyurl.com/8rkkduv'

iterative_count

Min: 0.117036 -> 0.156752: 1.34x slower Avg: 0.120802 -> 0.172218: 1.43x slower Significant (t=-34.92) Stddev: 0.00542 -> 0.00889: 1.6422x larger Timeline: b'http://tinyurl.com/9x9rtnk'

json_dump_v2

Min: 3.449868 -> 3.522645: 1.02x slower Avg: 3.467124 -> 3.541902: 1.02x slower Significant (t=-13.20) Stddev: 0.02701 -> 0.02960: 1.0959x larger Timeline: b'http://tinyurl.com/8bsz64a'

json_load

Min: 0.981740 -> 0.567611: 1.73x faster Avg: 0.986729 -> 0.572975: 1.72x faster Significant (t=128.95) Stddev: 0.01796 -> 0.01386: 1.2955x smaller Timeline: b'http://tinyurl.com/93txokx'

mako_v2

Min: 0.083660 -> 0.243323: 2.91x slower Avg: 0.084634 -> 0.247875: 2.93x slower Significant (t=-821.55) Stddev: 0.00193 -> 0.00400: 2.0737x larger Timeline: b'http://tinyurl.com/98n9fab'

meteor_contest

Min: 0.257992 -> 0.232116: 1.11x faster Avg: 0.262581 -> 0.236684: 1.11x faster Significant (t=14.31) Stddev: 0.00916 -> 0.00894: 1.0243x smaller Timeline: b'http://tinyurl.com/8tpjt43'

nbody

Min: 0.375414 -> 0.293685: 1.28x faster Avg: 0.379489 -> 0.299794: 1.27x faster Significant (t=42.71) Stddev: 0.00997 -> 0.00864: 1.1537x smaller Timeline: b'http://tinyurl.com/96aqtod'

normal_startup

Min: 0.360002 -> 0.593214: 1.65x slower Avg: 0.386755 -> 0.600625: 1.55x slower Significant (t=-134.28) Stddev: 0.01055 -> 0.00395: 2.6704x smaller Timeline: b'http://tinyurl.com/9td8pna'

nqueens

Min: 0.300390 -> 0.363904: 1.21x slower Avg: 0.304282 -> 0.368003: 1.21x slower Significant (t=-37.41) Stddev: 0.00813 -> 0.00888: 1.0920x larger Timeline: b'http://tinyurl.com/9zxyfcu'

pathlib

Min: 0.106088 -> 0.138693: 1.31x slower Avg: 0.107279 -> 0.139885: 1.30x slower Significant (t=-133.12) Stddev: 0.00256 -> 0.00290: 1.1324x larger Timeline: b'http://tinyurl.com/9llvj6a'

pidigits

Min: 0.351666 -> 0.341745: 1.03x faster Avg: 0.354743 -> 0.344146: 1.03x faster Significant (t=5.89) Stddev: 0.00965 -> 0.00829: 1.1643x smaller Timeline: b'http://tinyurl.com/8bkgrv4'

raytrace

Min: 1.547054 -> 1.641147: 1.06x slower Avg: 1.552614 -> 1.643716: 1.06x slower Significant (t=-286.42) Stddev: 0.00190 -> 0.00120: 1.5920x smaller Timeline: b'http://tinyurl.com/9bmnbsd'

regex_compile

Min: 0.494022 -> 0.537924: 1.09x slower Avg: 0.497904 -> 0.541971: 1.09x slower Significant (t=-18.23) Stddev: 0.01177 -> 0.01239: 1.0523x larger Timeline: b'http://tinyurl.com/cvdhrrm'

regex_effbot

Min: 0.065431 -> 0.073393: 1.12x slower Avg: 0.069753 -> 0.077338: 1.11x slower Significant (t=-10.61) Stddev: 0.00361 -> 0.00354: 1.0179x smaller Timeline: b'http://tinyurl.com/cupb89m'

regex_v8

Min: 0.071053 -> 0.081441: 1.15x slower Avg: 0.075075 -> 0.086167: 1.15x slower Significant (t=-12.44) Stddev: 0.00359 -> 0.00518: 1.4455x larger Timeline: b'http://tinyurl.com/d9ly6x3'

simple_logging

Min: 0.325386 -> 0.395093: 1.21x slower Avg: 0.330235 -> 0.399825: 1.21x slower Significant (t=-34.22) Stddev: 0.00952 -> 0.01077: 1.1317x larger Timeline: b'http://tinyurl.com/8sbqv85'

startup_nosite

Min: 0.082137 -> 0.453112: 5.52x slower Avg: 0.129994 -> 0.459361: 3.53x slower Significant (t=-276.85) Stddev: 0.01114 -> 0.00419: 2.6585x smaller Timeline: b'http://tinyurl.com/932mzal'

telco

Min: 0.810000 -> 0.010000: 81.00x faster Avg: 0.823600 -> 0.015200: 54.18x faster Significant (t=284.37) Stddev: 0.01946 -> 0.00505: 3.8556x smaller Timeline: http://chart.apis.google.com/chart?cht=lc&chs=700x400&chxt=x,y,x,y&chxr=1,0,1.9&chco=FF0000,0000FF&chdl=../cpython-2.7-wide/bin/python2.7|../cpython-3.3/bin/python3.3&chds=0,1.9&chd=t:0.9,0.84,0.81,0.82,0.82,0.82,0.82,0.82,0.81,0.82,0.82,0.82,0.82,0.82,0.81,0.82,0.82,0.82,0.82,0.82,0.81,0.82,0.82,0.82,0.82,0.81,0.82,0.82,0.82,0.82,0.82,0.81,0.82,0.82,0.82,0.82,0.82,0.81,0.81,0.82,0.82,0.82,0.82,0.81,0.82,0.89,0.89,0.86,0.81,0.82|0.02,0.01,0.02,0.01,0.02,0.01,0.02,0.01,0.02,0.02,0.01,0.02,0.02,0.01,0.02,0.01,0.02,0.01,0.02,0.01,0.02,0.02,0.01,0.02,0.01,0.02,0.01,0.02,0.01,0.02,0.01,0.01,0.02,0.01,0.02,0.01,0.02,0.01,0.02,0.01,0.02,0.01,0.02,0.01,0.02,0.01,0.02,0.01,0.02,0.01&chxl=0:|1|10|20|30|40|50|2:||Iteration|3:||Time+(secs)&chtt=telco

threaded_count

Min: 0.140653 -> 0.173500: 1.23x slower Avg: 0.152514 -> 0.270779: 1.78x slower Significant (t=-49.87) Stddev: 0.00605 -> 0.01564: 2.5837x larger Timeline: b'http://tinyurl.com/9w4u7el'

unpack_sequence

Min: 0.000077 -> 0.000067: 1.15x faster Avg: 0.000081 -> 0.000069: 1.18x faster Significant (t=1163.57) Stddev: 0.00000 -> 0.00000: 1.7412x larger Timeline: b'http://tinyurl.com/8qdcjcr'

The following not significant results are hidden, use -v to show them: html5lib, richards, silent_logging, spectral_norm. -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20121026/8cc1810d/attachment.html>

More information about the Python-Dev mailing list