(original) (raw)

Since Python 2.3 (with the introduction of the zipimport module) it's been sort-of possible to zip up the standard library. Modules/getpath.c:calculate\_path even adds a specific location ($prefix/lib/python33.zip) to sys.path if it exists to facilitate that. Or you can include the zipfile alongside an application that embeds Python, or even embed the zipfile in the same application. Actually setting things up is not quite that simple, though, at least on non-Windows: you need to know what to include in the zipfile (only .py, .pyc and .pyo files, no .so files), what to leave in the old location (os.py, or at least \*some file called os.py\*, needs to stay in $prefix/lib/python3.3) and how to deal with tests and modules that don't like the stdlib living in zipfiles -- and there's more of those than I expected. Also, depending on what else you want to put in the zipfile, you may have to be aware of zipimports limited implementation of zipfiles that involve various 32k-filecount and 2Gb-filesize limits. (And in case you're wondering, yes, we are doing this with Python 2.7 at Google to save space. And yes, hitting the 2Gb limit is quite possible for us.)

So with importlib going in, should we do something with zipimport as well? Its deficiencies can easily be fixed by reimpementing it in Python instead -- the zipfile module has long since fixed the same 32k/2Gb issues (reading signed instead of unsigned numbers) and actually supports zip64 extensions (to break the 64k-filecount and 4Gb-filesize limits in "normal" zipfiles.) Actually supporting zipping the stdlib then becomes a bit harder: the importlib bootstrapping would need to include the zipfile module. If we do that, it would be nice to actually support zipping the stdlib in the Python build: making a build target that actually does that, and runs the tests with it. However, this requires modification of a whole bunch of tests, for example ones that assume that stdlib modules (and the tests themselves!) have actual files you can open() as their \_\_file\_\_ attribute, and we'd need to run the testsuite with the stdlib as a zip to prevent new ones from sneaking in. Also, at that point the question becomes if we need a transparent interface for opening module sourcefiles or arbitrary files living in packages, that could grab things out of zipfiles (like setuptools has in... one of the modules) -- or other archives of course.

(And, yes, I'm zipping up the stdlib for Python 2.7 at Google, to reduce the impact on the aforementioned million of machines :)

--
Thomas Wouters <thomas@python.org>

Hi! I'm a .signature virus! copy me into your .signature file to help me spread!