[Python-Dev] Python startup time (original) (raw)

Nick Coghlan ncoghlan at gmail.com
Thu Jul 20 22:44:38 EDT 2017

On 21 July 2017 at 05:38, Cesare Di Mauro <cesare.di.mauro at gmail.com> wrote:

2017-07-20 19:23 GMT+02:00 Victor Stinner <victor.stinner at gmail.com>: 2017-07-20 19:09 GMT+02:00 Cesare Di Mauro <cesare.di.mauro at gmail.com>: > I assume that Python loads compiled (.pyc and/or .pyo) from the stdlib. That's something that also influences the startup time (compiling source vs loading pre-compiled modules).

My benchmark was "python3 -m perf command -- python3 -c pass": I don't explicitly remove .pyc files, I expect that Python uses prebuilt .pyc files from pycache. Victor OK, that should be the best case. An idea to improve the situation might be to find an alternative structure for .pyc/pyo files, which allows to (partially) "parallelize" their loading (not execution, of course), or at least speed-up the process. Maybe a GSoC project for some student, if no core dev has time to investigate it.

Unmarshalling the code object from disk generally isn't the slow part - it's the module level execution that takes time.

Using the typing module as an example, a full reload cycle takes almost 10 milliseconds:

$ python3 -m perf timeit -s "import typing; from importlib import reload" "reload(typing)" ..................... Mean +- std dev: 9.89 ms +- 0.46 ms

(Don't try timing "import typing" directly - the sys.modules cache amortises the cost down to being measured in nanoseconds, since you're effectively just measuring the speed of a dict lookup)

We can separately measure the cost of unmarshalling the code object:

$ python3 -m perf timeit -s "import typing; from marshal import loads; from importlib.util import cache_from_source; cache = cache_from_source(typing.file); data = open(cache, 'rb').read()[12:]" "loads(data)" ..................... Mean +- std dev: 286 us +- 4 us

Finding the module spec:

$ python3 -m perf timeit -s "from importlib.util import find_spec" "find_spec('typing')" ..................... Mean +- std dev: 69.2 us +- 2.3 us

And actually running the module's code (this includes unmarshalling the code object, but not calculating the import spec):

$ python3 -m perf timeit -s "import typing; loader_exec = typing.spec.loader.exec_module" "loader_exec(typing)" ..................... Mean +- std dev: 9.68 ms +- 0.43 ms

Cheers, Nick.

-- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20170721/ce1d38ef/attachment.html>

More information about the Python-Dev mailing list