| msg45624 - (view) |
Author: Martin v. Löwis (loewis) *  |
Date: 2004-03-23 00:10 |
| This patch uses sys.path_importer_cache to reduce the number of open calls, in the following way: - if the value in path_importer_cache is None, it stats the path to find out whether the file exists - it then puts True/False into path_importer_cache - if the value in path_importer_cache is False, the path entry is skipped on all imports - if the value is True, the stat call is skipped, and open calls for files in the directory are made. On Linux, this reduces the number of open calls for an empty script from 343 to 263. The startup-time (for 100 interpreter invocations) goes down by one percent (from 0.0819s to 0.08113s per invocation). |
|
|
| msg45625 - (view) |
Author: Raymond Hettinger (rhettinger) * |
Date: 2004-03-23 07:30 |
| Logged In: YES user_id=80475 I am surprised that making 25% fewer open calls doesn't save more than 1% in startup time. One other thought, I wonder if the timing of these changes is affected by the OS keeping recently loaded files in buffers so that disk access time not included. |
|
|
| msg45626 - (view) |
Author: Martin v. Löwis (loewis) * |
Date: 2004-03-23 15:43 |
| Logged In: YES user_id=21627 It's certainly the case that the system has cached all files needed for startup in memory, including the directory contents of all directories searched. OTOH, I assume that is the scenario in which people worry about startup time: high-frequency invocations of python. For a single invocation, it shouldn't matter much whether it takes 0.04s or 0.08s. |
|
|
| msg45627 - (view) |
Author: Georg Brandl (georg.brandl) * |
Date: 2006-02-20 10:42 |
| Logged In: YES user_id=1188172 Can this go into 2.5? |
|
|
| msg45628 - (view) |
Author: Martin v. Löwis (loewis) * |
Date: 2006-02-20 21:51 |
| Logged In: YES user_id=21627 Not sure. Anybody speaking in favour? against? |
|
|
| msg45629 - (view) |
Author: Georg Brandl (georg.brandl) * |
Date: 2006-02-25 19:55 |
| Logged In: YES user_id=849994 I'm very much for it. I haven't got too much RAM, and whenever I start a Python program (emerge being the most prominent example) after having worked heavily with e.g. graphics or VMware, I'm hit by the files Python's opening not being in the file cache anymore. |
|
|
| msg45630 - (view) |
Author: Georg Brandl (georg.brandl) * |
Date: 2006-05-25 18:47 |
| Logged In: YES user_id=849994 I reviewed this patch, in in consequence discovered a problem with the sys.path_hooks machinery, described in http://mail.python.org/pipermail/python-dev/2006-May/065173.html This patch fixes the problem and corrects the original patch to not set any sys.path_importer_cache entry to True or False when no import hooks are enabled (the p_loader argument to find_module is NULL then). |
|
|
| msg45631 - (view) |
Author: Neal Norwitz (nnorwitz) * |
Date: 2006-05-26 06:38 |
| Logged In: YES user_id=33168 Without looking at the patch impl, I'm +1 on the idea of reducing stat/open calls. On NFS this is a huge time sync. |
|
|
| msg45632 - (view) |
Author: Martin v. Löwis (loewis) * |
Date: 2006-05-26 17:35 |
| Logged In: YES user_id=21627 Your revised patch looks fine to me, so please apply. |
|
|
| msg45633 - (view) |
Author: Neal Norwitz (nnorwitz) * |
Date: 2006-05-27 03:48 |
| Logged In: YES user_id=33168 Georg, didn't you check this in or was that a diff patch? |
|
|
| msg45634 - (view) |
Author: Georg Brandl (georg.brandl) * |
Date: 2006-05-27 09:23 |
| Logged In: YES user_id=849994 Yes, I indeed checked this in, in rev. 46372. Thanks for reminding me, Neal. |
|
|