(original) (raw)

On Thu, Sep 15, 2016 at 5:57 PM Victor Stinner <victor.stinner@gmail.com> wrote:

2016-09-15 10:02 GMT+02:00 INADA Naoki <songofacandy@gmail.com>:
\> In my environ:
\>
\> \~/local/python-master/bin/python3 -m timeit -s "d =
\> dict.fromkeys(range(10\*\*6))" 'list(d)'

Stooooop! Please stop using timeit, it's lying!

\* You must not use the minimum but average or median
\* You must run a microbenchmark in multiple processes to test
different randomized hash functions and different memory layouts

In short: you should use my perf module.
http://perf.readthedocs.io/en/latest/cli.html#timeit

I'm sorry. Changing habit is bit difficult. I'll use it in next time.

I ran microbench 3\~5 times and confirm the result is stable before posting result.

And when difference is smaller than 10%, I don't believe the result.

The memory layout and the hash function have a major important on such
microbenchmark:
https://haypo.github.io/journey-to-stable-benchmark-average.html

In this microbench, hash randomization is not important, because key

of dict is int.

(It means iterating dict doesn't cause random memory access in old dict

implementation too.)

\> Both Python is built without neither \`--with-optimizations\` or \`make
\> profile-opt\`.

That's bad :-) For most reliable benchmarks, it's better to use
LTO+PGO compilation.

LTO+PGO may make performance of \`git pull && make\` unstable.

PGO clean build takes tooo long time for such a quick benchmark.

So I don't want to use PGO in such a quick benchmark.

And Python doesn't provide way to use LTO without PGO....