Using emacs' unexec to speed Python startup (was Re: [Python-Dev] Startup time) (original) (raw)

Daniel Berlin dberlin@dberlin.org
Sun, 18 May 2003 21:56:58 -0400

On Sunday, May 18, 2003, at 09:22 PM, Jeff Epler wrote:

On Fri, May 16, 2003 at 03:09:31PM -0400, Barry Warsaw wrote:

Skip, you're going about this all wrong. We already have the technology to start Python up blazingly fast. All you have to do is port XEmacs's unexec code. Then you load up Python with all the modules you think you're going to need, unexec it, then the next time it starts up like lightening. Disk space is cheap! I gave it a try, starting with 2.3b1 and using FSF Emacs 21.3's unexelf.c.

XEmacs has a portable undumper, IIRC.

An unexec'd binary loads faster than 'python -S -c pass', and seems to work properly with two exceptions and a few limitations.

The only change to Python is in main(): I use mallopt() to force all allocations to go through brk() instead of through mmap(), because unexec doesn't support mmap'd memory. I also used Modules/Setup.local to make some normally-shared modules not shared (for the same reason). dump.py loads the requested modules (- forces the module to not be found) and then calls unexec(), producing a new binary with the given name. $ time ./python -S -c pass # best 'real' of 5 runs real 0m0.054s user 0m0.040s sys 0m0.010s $ time ./python -c 'import cgi' # best 'real' of 5 runs real 0m0.127s user 0m0.110s sys 0m0.010s $ strace -e open ./python -c 'import cgi' 2>&1 | grep -v ENOENT | wc -l 88 $ ./python dump.py cgipython -ssl cgi $ time ./cgipython -c 'import cgi' # best 'real' of 5 runs real 0m0.039s user 0m0.020s sys 0m0.020s $ strace -e open ./cgipython -c 'import cgi' 2>&1 | grep -v ENOENT | wc -l 9 $ ./python dump.py dython -rwxrwxr-x 1 jepler jepler 4983713 May 18 19:42 cgipython -rwxrwxr-x 1 jepler jepler 3603737 May 18 19:39 python -rwxrwxr-x 1 jepler jepler 4541345 May 18 19:55 dython (a minimal unexec'd python is about 90k bigger than the regular Python binary) I'm running the test suite now .. it hangs in testsignal for some reason. testthread seems to hang too, which may be related. (but testthreading completes?) $ ./dython Lib/test/regrtest.py -x testsignal -x testthread [...] 225 tests OK. 26 tests skipped: testaepack testal testbsddb3 testbz2 testcd testcl testcurses testemailcodecs testgl testimgfile testlinuxaudiodev testmacfs testmacostools testnis testnormalization testossaudiodev testpep277 testplistlib testscriptpackages testsocketssl testsocketserver testsunaudiodev testtimeout testurllibnet testwinreg testwinsound 1 skip unexpected on linux2: testbz2 Well, if it worked right it'd sure be interesting. OTOH, unexelf.c is GPL'd and there's also the nightmare of different unex* for different platforms. Like I said, xemacs has a "portable" undumper. --Dan