[Python-Dev] Investigating time for import requests (original) (raw)

Nick Coghlan ncoghlan at gmail.com
Mon Oct 2 03:39:11 EDT 2017

On 2 October 2017 at 15:13, Raymond Hettinger <raymond.hettinger at gmail.com> wrote:

On Oct 1, 2017, at 7:34 PM, Nathaniel Smith <njs at pobox.com> wrote:

In principle re.compile() itself could be made lazy -- return a regular exception object that just holds the string, and then compiles and caches it the first time it's used. Might be tricky to do in a backwards compatibility way if it moves detection of invalid regexes from compile time to use time, but it could be an opt-in flag. ISTM that someone writing re.compile(pattern) is explicitly saying they want the regex to be pre-compiled. For cache on first-use, we already have a way to do that with re.search(pattern, some string) which compiles and then caches. What would be more interesting would be to have a way to save the compiled regex in a pyc file so that it can be restored on load rather than recomputed. Also, we should remind ourselves that making more and more things lazy is a false optimization unless those things never get used. Otherwise, all we're doing is ending the timing before all the relevant work is done. If the lazy object does get used, we've made the actual total execution time worse (because of the overhead of the lazy evaluation logic).

Right, and I think the approach Inada-san took here is a good example of how to do that effectively (there are a lot of command line scripts and other startup-sensitive operations that will include an "import requests", but not directly import any of the other modules in its dependency tree, so "What requests uses" can identify a useful set of avoidable imports. A Flask "Hello world" app could likely provide another such sample, as could some example data analysis notebooks).

Cheers, Nick.

-- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia

More information about the Python-Dev mailing list