[Python-Dev] os.walk() is going to be fast with scandir (original) (raw)

Ben Hoyt benhoyt at gmail.com
Sat Aug 9 18:43:01 CEST 2014

Just thought I'd share some of my excitement about how fast the all-C version [1] of os.scandir() is turning out to be.

Below are the results of my scandir / walk benchmark run with three different versions. I'm using an SSD, which seems to make it especially faster than listdir / walk. Note that benchmark results can vary a lot, depending on operating system, file system, hard drive type, and the OS's caching state.

Anyway, os.walk() can be FIFTY times as fast using os.scandir().

Old ctypes implementation of scandir in scandir.py:

C:\work\scandir>\work\python\cpython\python benchmark.py -r Using slower ctypes version of scandir os.walk took 1.144s, scandir.walk took 0.060s -- 19.2x as fast

Existing "half C" implementation of scandir in _scandir.c:

C:\work\scandir>\Python34-x86\python.exe benchmark.py -r Using fast C version of scandir os.walk took 1.160s, scandir.walk took 0.042s -- 27.6x as fast

New "all C" os.scandir implementation in posixmodule.c:

C:\work\scandir>\work\python\cpython\python benchmark.py -r Using Python 3.5's builtin os.scandir() os.walk took 1.141s, scandir.walk took 0.022s -- 53.0x as fast

[1] Work in progress implementation as part of Python 3.5's posixmodule.c available here: https://github.com/benhoyt/scandir/blob/master/posixmodule.c

-Ben

More information about the Python-Dev mailing list