[Python-Dev] PEP 441 - Improving Python ZIP Application Support (original) (raw)

Barry Warsaw barry at python.org
Tue Feb 17 20:00:38 CET 2015

On Feb 15, 2015, at 05:21 PM, Thomas Wouters wrote:

FWIW, we have a similar tool to 'pyzaa' at Google, although a lot older (it has a pure-python implementation of zipimport, because it had to work with Python 2.2 back in the day) that does support extension modules and other shared libraries (by extracting them to disk on program startup.) We're also working to replace that by loading the extension modules directly from the archive (using dlopenwithoffset as described by ppluzhnikov in https://sourceware.org/bugzilla/showbug.cgi?id=11767, although it should also be possible using dlopenehdr/dlopenphdr), which requires that extension modules are stored uncompressed (simple) and page-aligned (harder, as the zipfile.ZipFile class doesn't directly support page-aligning anything, but it turns out it's easy to hack around by overriding writecheck :P) And yes, we really are in a position to modify glibc to make our life distributing Python applications easier, internally; the old code involves shell and Python bootstrap code that tries very hard to avoid race conditions and security problems, which is a pain to maintain.

So, let's add this to zipimport so we can remove an unfortunate limitation and avoid everyone having to reinvent the wheel!

Oh, and the dlopen extension sound pretty cool. Can't wait until Google gets that into glibc so us mere mortals can use it. Seems reasonable to add this to zipimport too, with whatever feature guards are necessary. You'll want to give your future self an opportunity to trot out the keys to the time machine. :)

It might be nice to consider those use-cases in pyzapp as well, especially once the glibc feature is released. It requires some fairly big changes to zipimport (I ended up rewriting the whole thing) but we can easily opensource the code, if there's any interest at all.

+1

I'd be happy to discuss this in more detail at PyCon, at which time we should be deploying code internally using all of this.

+1

Cheers, -Barry

More information about the Python-Dev mailing list