[Python-Dev] PyDict_SetItem hook (original) (raw)

M.-A. Lemburg mal at egenix.com
Fri Apr 3 19:04:36 CEST 2009

On 2009-04-03 18:06, Thomas Wouters wrote:

On Fri, Apr 3, 2009 at 11:27, Antoine Pitrou <solipsis at pitrou.net> wrote:

Thomas Wouters <thomas python.org> writes:

Pystone is pretty much a useless benchmark. If it measures anything, it's the speed of the bytecode dispatcher (and it doesn't measure it particularly well.) PyBench isn't any better, in my experience. I don't think pybench is useless. It gives a lot of performance data about crucial internal operations of the interpreter. It is of course very little real-world, but conversely makes you know immediately where a performance regression has happened. (by contrast, if you witness a regression in a high-level benchmark, you still have a lot of investigation to do to find out where exactly something bad happened) Really? Have you tried it? I get at least 5% noise between runs without any changes. I have gotten results that include negative run times.

On which platform ? pybench 2.0 works reasonably well on Linux and Windows, but of course can't do better than the timers available for those platforms. If you have e.g. NTP running and it uses wall clock timers, it is possible that you get negative round times. If you don't and still get negative round times, you have to change the test parameters (see below).

And yes, I tried all the different settings for calibration runs and timing mechanisms. The tests in PyBench are not micro-benchmarks (they do way too much for that), they don't try to minimize overhead or noise,

That is not true. They were written as micro-benchmarks and adjusted to have a high signal-noise ratio. For some operations this isn't easy to do, but I certainly tried hard to get the overhead low (note that the overhead is listed in the output).

That said, please keep in mind that the settings in pybench were last adjusted some years ago to have the tests all run in more or less the same wall clock time. CPUs have evolved a lot since then and this shows.

but they are also not representative of real-world code.

True and they never were meant for that, since I was frustrated by other benchmarks at the time and the whole approach in general.

Each of the tests checks one specific aspect of Python. If your application happens to use a lot of dictionary operations, you'll be mostly interested in those. If you do a lot of simple arithmetic, there's another test for that.

On top of that the application is written to be easily extensible, so it's easy to add new tests specific to whatever application space you're after.

That doesn't just mean "you can't infer the affected operation from the test name", but "you can't infer anything." You can just be looking at differently borrowed runtime. I have in the past written patches to Python that improved every micro-benchmark and every real-world measurement I made, except PyBench. Trying to pinpoint the slowdown invariably lead to tests that did too much in the measurement loop, introduced too much noise in the "calibration" run or just spent their time in the measurement loop on doing setup and teardown of the test.

pybench calibrates itself to remove that kind of noise from the output. Each test has a .calibrate() method which does all the setup and tear down minus the actual benchmark operations.

If you get wrong numbers, try adjusting the parameters and add more "packets" of operations. Don't forget to adjust the version number to not compare apples and orange, though.

Perhaps it's time to readjust the pybench parameters to todays CPUs.

-- Marc-Andre Lemburg eGenix.com

Professional Python Services directly from the Source (#1, Apr 03 2009)

Python/Zope Consulting and Support ... http://www.egenix.com/ mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/

2009-03-19: Released mxODBC.Connect 1.0.1 http://python.egenix.com/

::: Try our new mxODBC.Connect Python Database Interface for free ! ::::

eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/

More information about the Python-Dev mailing list