[Python-Dev] CHUNKSIZE (original) (raw)

Guido van Rossum guido@python.org
Sat, 27 Apr 2002 17:59:51 -0400

Yeah, you're probably right. Hopefully someone knowledgeable on the subject will enlighten us.

I bet you wouldn't be able to detect any performance improvement if the xreadlines() buffer were increased in size. See the comment in the source:

/* A larger chunk size doesn't seem to make a difference */ #define CHUNKSIZE 8192

It does beg the question though: Why isn't readline mmap'd? Would that be more efficient?

(a) File objects are not just used for disk files, but also for sockets (via socket.makefile()), pipes, etc.

(b) Mmap APIs vary widely across systems -- we've only got mmap support on Unix and Windows.

All this can be addressed by rewriting the I/O system, but that's a huge job that nobody has volunteered to tackle yet. (It's particularly painful because it has to work cross-platform, and beyond stdio there's nothing we can rely on cross-platform.)

--Guido van Rossum (home page: http://www.python.org/~guido/)