[Python-Dev] os.walk() is going to be fast with scandir (original) (raw)

Akira Li 4kir4.1i at gmail.com
Mon Aug 11 17:26:29 CEST 2014

Armin Rigo <arigo at tunes.org> writes:

On 10 August 2014 08:11, Larry Hastings <larry at hastings.org> wrote:

A small tip from my bzr days - cd into the directory before scanning it

I doubt that's permissible for a library function like os.scandir(). Indeed, chdir() is notably not compatible with multithreading. There would be a non-portable but clean way to do that: the functions openat() and fstatat(). They only exist on relatively modern Linuxes, though.

There is os.fwalk() that could be both safer and faster than os.walk(). It yields rootdir fd that can be used by functions that support dir_fd parameter, see os.supports_dir_fd set. They use *at() functions under the hood.

os.fwalk() could be implemented in terms of os.scandir() if the latter would support fd parameter like os.listdir() does (be in os.supports_fd set (note: it is different from os.supports_dir_fd)).

Victor Stinner suggested [1] to allow scandir(fd) but I don't see it being mentioned in the pep 471 [2]: it neither supports nor rejects the idea.

[1] https://mail.python.org/pipermail/python-dev/2014-July/135283.html [2] http://legacy.python.org/dev/peps/pep-0471/

-- Akira

More information about the Python-Dev mailing list