[Python-Dev] More optimisation ideas (original) (raw)

Oscar Benjamin oscar.j.benjamin at gmail.com
Sat Jan 30 08:54:54 EST 2016

On 30 January 2016 at 03:48, Steve Dower <steve.dower at python.org> wrote:

It doesn't currently end up on disk. Some tables are partially or completely stored on disk as Python source code (some are partially generated from simple rules), but others are generated by inverting those. That process takes time that could be avoided by storing the generated tables, and storing all of it in a format that doesn't require parsing, compiling and executing (such as a native array). Potentially it could be a win all around if we stopped including the (larger) source files, but that doesn't seem like a good idea for maintaining portability to other implementations. The main thought is making the compiler binary bigger to avoid generating encoding tables at startup.

When I last tried to profile startup on Windows (I haven't used Windows for some time now) it seemed that the time was totally dominated by file system access. Essentially the limiting factor was the inordinate number of stat calls and small file accesses. Although this was probably Python 2.x which may not import those particular modules and maybe it depends on virus scanner software etc.

Things may have changed now but I concluded that substantive gains could only come from improving FS access. Perhaps something like zipping up the standard library would see a big improvement.

-- Oscar

More information about the Python-Dev mailing list