[Python-Dev] iterzip() (original) (raw)

Tim Peters tim.one@comcast.net
Mon, 29 Apr 2002 01:05:15 -0400

[Raymond Hettinger]

... I just timed it and am shocked. iterzip() has exactly the same code as zip() except for the final append result to list. So, I expected only a microscopic speed-up. I don't see where the huge performance improvement came from.

Well, in the zip case you're allocating 4 million bytes worth of integers, 4 bytes at a time, and don't get to reuse any of the space for the duration. Likewise you're allocating space for a million 3-element tuples, and 4MB worth of contiguous vector 4 bytes at a time. That's all a real drag. Cute: I can make the Python genzip run faster than the builtin zip with "the usual" speed tricks:

from time import clock as now

def run(f, iterables): start = now() for tup in f(*iterables): pass return now() - start

def genzip(*iterables): getters = [iter(i).next for i in iterables] t = tuple while 1: yield t([g() for g in getters])

n = 1000000 for f in [zip, genzip]: print run(f, [xrange(n), xrange(n), xrange(n)]), f.name

On my box under current CVS, that yields

C:\Code\python\PCbuild>python -O temp.py 12.3587522419 zip 10.4161551484 genzip

Better, I can get genzip down to 8.34 seconds by skipping the tuple() call -- in real life, it doesn't really matter whether it returns a tuple or a list.

One reason zip is faster for me than for you is that pymalloc is enabled by default in 2.3, and can likely pump out space for a million small tuples significantly faster than your platform malloc can do it.

First timing: ---------------------- 19.439999938 zip 1.43000006676 iterzip 30.1000000238 genzip

If only I had a use for iterzip .