(original) (raw)
On 21 July 2017 at 05:38, Cesare Di Mauro <cesare.di.mauro@gmail.com> wrote:
$ python3 -m perf timeit -s "import typing; from marshal import loads; from importlib.util import cache\_from\_source; cache = cache\_from\_source(typing.\_\_file\_\_); data = open(cache, 'rb').read()\[12:\]" "loads(data)"
.....................
Mean +- std dev: 286 us +- 4 us
2017-07-20 19:23 GMT+02:00 Victor Stinner <victor.stinner@gmail.com>:2017-07-20 19:09 GMT+02:00 Cesare Di Mauro <cesare.di.mauro@gmail.com>:
\> I assume that Python loads compiled (.pyc and/or .pyo) from the stdlib. That's something that also influences the startup time (compiling source vs loading pre-compiled modules).
My benchmark was "python3 -m perf command -- python3 -c pass": I don't
explicitly remove .pyc files, I expect that Python uses prebuilt .pyc
files from \_\_pycache\_\_.
Victor
OK, that should be the best case.An idea to improve the situation might be to find an alternative structure for .pyc/pyo files, which allows to (partially) "parallelize" their loading (not execution, of course), or at least speed-up the process. Maybe a GSoC project for some student, if no core dev has time to investigate it.
Unmarshalling the code object from disk generally isn't the slow part - it's the module level execution that takes time.
Using the typing module as an example, a full reload cycle takes almost 10 milliseconds:
$ python3 -m perf timeit -s "import typing; from importlib import reload" "reload(typing)"
.....................
Mean +- std dev: 9.89 ms +- 0.46 ms
$ python3 -m perf timeit -s "import typing; from importlib import reload" "reload(typing)"
.....................
Mean +- std dev: 9.89 ms +- 0.46 ms
(Don't try timing "import typing" directly - the sys.modules cache amortises the cost down to being measured in nanoseconds, since you're effectively just measuring the speed of a dict lookup)
We can separately measure the cost of unmarshalling the code object:
$ python3 -m perf timeit -s "import typing; from marshal import loads; from importlib.util import cache\_from\_source; cache = cache\_from\_source(typing.\_\_file\_\_); data = open(cache, 'rb').read()\[12:\]" "loads(data)"
.....................
Mean +- std dev: 286 us +- 4 us
Finding the module spec:
$ python3 -m perf timeit -s "from importlib.util import find\_spec" "find\_spec('typing')"
.....................
Mean +- std dev: 69.2 us +- 2.3 us
$ python3 -m perf timeit -s "from importlib.util import find\_spec" "find\_spec('typing')"
.....................
Mean +- std dev: 69.2 us +- 2.3 us
And actually running the module's code (this includes unmarshalling the code object, but \*not\* calculating the import spec):
$ python3 -m perf timeit -s "import typing; loader\_exec = typing.\_\_spec\_\_.loader.exec\_module" "loader\_exec(typing)"
.....................
Mean +- std dev: 9.68 ms +- 0.43 ms
$ python3 -m perf timeit -s "import typing; loader\_exec = typing.\_\_spec\_\_.loader.exec\_module" "loader\_exec(typing)"
.....................
Mean +- std dev: 9.68 ms +- 0.43 ms
Cheers,
Nick.
--
Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia