[Python-Dev] Python startup time (original) (raw)

Gregory Szorc gregory.szorc at gmail.com
Wed May 2 12:42:55 EDT 2018

On Tue, May 1, 2018 at 11:55 PM, Ray Donnelly <mingw.android at gmail.com> wrote:

Is your Python interpreter statically linked? The Python 3 ones from the anaconda distribution (use Miniconda!) are for Linux and macOS and that roughly halved our startup times.

My Python interpreters use a shared library. I'll definitely investigate the performance of a statically-linked interpreter.

Correct me if I'm wrong, but aren't there downsides with regards to C extension compatibility to not having a shared libpython? Or does all the packaging tooling "just work" without a libpython? (It's possible I have my wires crossed up with something else regarding a statically linked Python.)

On Wed, May 2, 2018 at 2:26 AM, Victor Stinner <vstinner at redhat.com> wrote:

What do you propose to make Python startup faster?

That's a very good question. I'm not sure I'm able to answer it because I haven't dug too much into CPython's internals much farther than what is required to implement C extensions. But I can share insight from what the Mercurial project has collectively learned.

As I wrote in my previous emails, many Python core developers care of the startup time and we are working on making it faster. INADA Naoki added -X importtime to identify slow imports and understand where Python spent its startup time.

-X importtime is a great start! For a follow-up enhancement, it would be useful to see what aspects of import are slow. Is it finding modules (involves filesystem I/O)? Is it unmarshaling pyc files? Is it executing the module code? If executing code, what part is slow? Inline statements/expressions? Compiling types? Printing the microseconds it takes to import a module is useful. But it only gives me a general direction: I want to know what parts of the import made it slow so I know if I should be focusing on code running during module import, slimming down the size of a module, eliminating the module import from fast paths, pursuing alternative module importers, etc.

Recent example: Barry Warsaw identified that pkgresources is slow and added importlib.resources to Python 3.7: https://docs.python.org/dev/library/importlib.html#module- importlib.resources Brett Cannon is also working on a standard solution for lazy imports since many years: https://pypi.org/project/modutil/ https://snarky.ca/lazy-importing-in-python-3-7/

Mercurial has used lazy module imports for years. On 2.7.14, it reduces hg version from 160ms to ~55ms (34% of original). On Python 3, we're using importlib.util.LazyLoader and it reduces hg version on 3.7 from 245ms to ~120ms (49% of original). I'm not sure why Python 3's built-in module importer doesn't yield the speedup that our custom Python 2 importer does. One explanation is our custom importer is more advanced than importlib. Another is that Python 3's import mechanism is slower (possibly due to being written in Python instead of C). We haven't yet spent much time optimizing Mercurial for Python 3: our immediate goal is to get it working first. Given the startup performance problem on Python 3, it is only a matter of time before we dig into this further.

It's worth noting that lazy module importing can be undone via common patterns. Most commonly, from foo import X. It's really difficult to implement a proper object proxy. Mercurial's lazy importer gives up in this case and imports the module and exports the symbol. (But if the imported module is a package, we detect that and make the module exports proxies to a lazy module.)

Another common undermining of the lazy importer is code that runs during import time module exec that accesses an attribute. e.g.

import foo

class myobject(foo.Foo):
    pass

Mercurial goes out of its way to avoid these patterns so modules can be delay imported as much as possible. As long as import times are problematic, it would be helpful if the standard library adopted similar patterns. Although I recognize there are backwards compatibility concerns that tie your hands a bit.

Nick Coghlan is working on the C API to configure Python startup: PEP 432. When it will be ready, maybe Mercurial could use a custom Python optimized for its use case.

That looks great!

The direction Mercurial is going in is that hg will likely become a Rust binary (instead of a #!python script) that will use an embedded Python interpreter. So we will have low-level control over the interpreter via the C API. I'd also like to see us distribute a copy of Python in our official builds. This will allow us to take various shortcuts, such as not having to probe various sys.path entries since certain packages can only exist in one place. I'd love to get to the state Google is at where they have self-contained binaries with ELF sections containing Python modules. But that requires a bit of very low-level hacking. We'll likely have a Rust binary (that possibly static links libpython) and a separate JAR/zip-like file containing resources.

But many people obtain Python via their system package manager and no matter how hard we scream that Mercurial is a standalone application, they will configure their packages to link against the system libpython and use the system Python's standard library. This will potentially undo many of our startup time wins.

IMHO Python import system is inefficient. We try too many alternative names. Example with Python 3.8 $ ./python -vv: >>> import dontexist # trying /home/vstinner/prog/python/master/dontexist.cpython-38dm- x8664-linux-gnu.so # trying /home/vstinner/prog/python/master/dontexist.abi3.so # trying /home/vstinner/prog/python/master/dontexist.so # trying /home/vstinner/prog/python/master/dontexist.py # trying /home/vstinner/prog/python/master/dontexist.pyc # trying /home/vstinner/prog/python/master/Lib/dontexist.cpython- 38dm-x8664-linux-gnu.so # trying /home/vstinner/prog/python/master/Lib/dontexist.abi3.so # trying /home/vstinner/prog/python/master/Lib/dontexist.so # trying /home/vstinner/prog/python/master/Lib/dontexist.py # trying /home/vstinner/prog/python/master/Lib/dontexist.pyc # trying /home/vstinner/prog/python/master/build/lib.linux-x8664- 3.8-pydebug/dontexist.cpython-38dm-x8664-linux-gnu.so # trying /home/vstinner/prog/python/master/build/lib.linux-x8664- 3.8-pydebug/dontexist.abi3.so # trying /home/vstinner/prog/python/master/build/lib.linux-x8664- 3.8-pydebug/dontexist.so # trying /home/vstinner/prog/python/master/build/lib.linux-x8664- 3.8-pydebug/dontexist.py # trying /home/vstinner/prog/python/master/build/lib.linux-x8664- 3.8-pydebug/dontexist.pyc # trying /home/vstinner/.local/lib/python3.8/site-packages/dontex ist.cpython-38dm-x8664-linux-gnu.so # trying /home/vstinner/.local/lib/python3.8/site-packages/dontex ist.abi3.so # trying /home/vstinner/.local/lib/python3.8/site-packages/dontexist.so # trying /home/vstinner/.local/lib/python3.8/site-packages/dontexist.py # trying /home/vstinner/.local/lib/python3.8/site-packages/dontexist.pyc Traceback (most recent call last): File "", line 1, in File "", line 983, in findandload File "", line 965, in findandloadunlocked ModuleNotFoundError: No module named 'dontexist' Why do we still check for the .pyc file outside pycache directories? Why do we have to check for 3 different names for .so files?

Yes, I also cringe every time I trace Python's system calls and see these needless stats and file opens. Unless Python adds the ability to tell the import mechanism what type of module to import, Mercurial will likely modify our custom importer to only look for specific files. We do provide pure Python modules for modules that have C implementations. But we have code that ensures that the C version is loaded for certain Python configurations because we don't want users accidentally using the non-C modules and then complaining about Mercurial's performance! We already denote the set of modules backed by C. What we're missing (but is certainly possible to implement) is code that limits the module finding search depending on whether the module is backed by Python or C. But this only really works for Mercurial's modules: we don't really know what the standard library is doing and coding assumptions into Mercurial about standard library behavior feels dangerous.

If we ship our own Python distribution, we'll likely have a jar-like file containing all modules. Determining which file to load will read an in-memory file index and not require any expensive system calls to look for files.

Does Mercurial need all directories of sys.path?

No and yes. Mercurial by itself can get by with just the standard library and Mercurial's own packages. But extensions change everything. An extension could modify sys.path though. So limiting sys.path inside Mercurial is somewhat reasonable. Although it's definitely unexpected for a Python application to be removing entries from sys.path when the application starts.

What's the status of the "system python" project? :-) I also would prefer Python without the site module. Can we rewrite this module in C maybe? Until recently, the site module was needed on Python to create the "mbcs" encoding alias. Hopefully, the feature has been removed into Lib/encodings/init.py (new private aliasmbcs() function).

I also lament the startup time effects of site.py. When hg is a Rust binary, we will almost certainly skip site.py and manually perform any required actions that it was performing.

Python 3.7b3+: $ python3.7 -X importtime -c pass import time: self [us] | cumulative | imported package import time: 95 | 95 | zipimport import time: 589 | 589 | frozenimportlibexternal import time: 67 | 67 | codecs import time: 498 | 565 | codecs import time: 425 | 425 | encodings.aliases import time: 641 | 1629 | encodings import time: 228 | 228 | encodings.utf8 import time: 143 | 143 | signal import time: 335 | 335 | encodings.latin1 import time: 58 | 58 | abc import time: 265 | 322 | abc import time: 298 | 619 | io import time: 69 | 69 | stat import time: 196 | 265 | stat import time: 169 | 169 | genericpath import time: 336 | 505 | posixpath import time: 1190 | 1190 | collectionsabc import time: 600 | 2557 | os import time: 223 | 223 | sitebuiltins import time: 214 | 214 | sitecustomize import time: 74 | 74 | usercustomize import time: 477 | 3544 | site

As for things Python could do to make things better, one idea is for "package bundles." Instead of using .py, .pyc, .so, etc files as separate files on the filesystem, allow Python packages to be distributed as standalone "archive" files. Like Java's jar files. This has the advantage that there is only a single place to look for files in a given Python package. And since the bundle is immutable, you can index it so imports don't need to touch the filesystem to discover what is present: you do a quick memory lookup and jump straight to the available file. If you go this route, please don't require the use of zlib for file compression, as zlib is painfully slow compared to alternatives like lz4 and zstandard.

I know this kinda/sorta exists with zipimporter. But zipimporter uses zlib (slow) and only allows .py/.pyc files. And I think some Python application distribution tools have also solved this problem. I'd really like to see a proper/robust solution in Python itself. Along that vein, it would be really nice if the "standalone Python application" story were a bit more formalized. From my perspective, it is insanely difficult to package and distribute an application that happens to use Python. It requires vastly different solutions for different platforms. I want to declare a minimal boilerplate somewhere (perhaps in setup.py) and run a command that produces an as-self-contained-as-possible application complete with platform-native installers. Presumably such a self-contained application could take many shortcuts with regards to process startup and mitigate this general problem. Again, Mercurial is trending in the direction of making hg a Rust binary and distributing its own Python. Since we have to solve this packaging+distribution problem on multiple platforms, I'll try to keep an eye towards making whatever solution we concoct reusable by other projects. -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20180502/4a0516a4/attachment.html>

More information about the Python-Dev mailing list