[Python-Dev] Python Benchmarks (original) (raw)

M.-A. Lemburg mal at egenix.com
Fri Jun 2 12:26:02 CEST 2006

Fredrik Lundh wrote:

M.-A. Lemburg wrote:

Of course, but then changes to try-except logic can interfere with the performance of setting up method calls. This is what pybench then uncovers. I think the only thing PyBench has uncovered is that you're convinced that it's always right, and everybody else is always wrong, including people who've spent decades measuring performance, and the hardware in your own computer.

Oh, come on. You know that's not true and I'm trying to understand what is causing your findings, but this is difficult, since you're not providing enough details. E.g. the output of pybench showing the timing results would help a lot.

I would also like to reproduce your findings. Do you have two revision numbers in svn which I could use for this ?

See above (or the code in pybench.py). t1-t0 is usually around 20-50 seconds: what machines are you using? using the default parameters, the entire run takes about 50 seconds on the slowest machine I could find...

If the whole suite runs in 50 seconds, the per-test run-times are far too small to be accurate. I usually adjust the warp factor so that each round takes 50 seconds.

Looks like I have to revisit the default parameters and update the doc-strings. I'll do that when I add the new timers.

Could you check whether you still see the same results with running with "pybench.py -w 1" ?

that's not a very good idea, given how getprocesstime tends to be implemented on current-era systems (google for "jiffies")... but it definitely explains the bogus subtest results I'm seeing, and the "magic hardware" behaviour you're seeing. That's exactly the reason why tests run for a relatively long time - to minimize these effects. Of course, using wall time make this approach vulnerable to other effects such as current load of the system, other processes having a higher priority interfering with the timed process, etc. since process time is sampled, not measured, process time isn't exactly in- vulnerable either. it's not hard to imagine scenarios where you end up being assigned only a small part of the process time you're actually using, or cases where you're assigned more time than you've had a chance to use. afaik, if you want true performance counters on Linux, you need to patch the operating system (unless something's changed in very recent versions). I don't think that sampling errors can explain all the anomalies we've been seeing, but I'd wouldn't be surprised if a high-resolution wall time clock on a lightly loaded multiprocess system was, in practice, more reliable than sampled process time on an equally loaded system.

That's why the timers being used by pybench will become a parameter that you can then select to adapt pybench it to the OS your running pybench on.

Note that time.clock, the current default timer in pybench, is a high accuracy wall-clock timer on Windows, so it should demonstrate similar behavior to timeit.py, even more so, since your using warp 20 and thus a similar timing strategy as that of timeit.py.

I suspect that the calibration step is causing problems.

Steve added a parameter to change the number of calibration runs done per test: -C n. The default is 20.

-- Marc-Andre Lemburg eGenix.com

Professional Python Services directly from the Source (#1, Jun 02 2006)

Python/Zope Consulting and Support ... http://www.egenix.com/ mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/

2006-07-03: EuroPython 2006, CERN, Switzerland 30 days left

::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! ::::

More information about the Python-Dev mailing list