[Python-Dev] startup time repeated? why not daemon (original) (raw)

Chris Jerdonek chris.jerdonek at gmail.com
Sun Jul 23 19:48:23 EDT 2017

On Sun, Jul 23, 2017 at 5:57 AM, Victor Stinner <victor.stinner at gmail.com> wrote:

We already did that. See bootlocale for example. (Maybe also collecctionsabc?)

I was asking more in the context of recommended practices for third-party developers, as Nick mentioned earlier, because it's not a strategy I've ever seen mentioned (and common practice is to group only by functionality).

It's good to know re: locale and collections though. Incidentally, from the issue thread it doesn't look like _bootlocale was motivated primarily by startup time, but _collections_abc was:

locale: http://bugs.python.org/issue9548 collections.abc: http://bugs.python.org/issue19218

--Chris

Victor Le 22 juil. 2017 07:20, "Chris Jerdonek" <chris.jerdonek at gmail.com> a écrit :

On Fri, Jul 21, 2017 at 9:52 AM, Brett Cannon <brett at python.org> wrote: > On Thu, 20 Jul 2017 at 22:11 Chris Jerdonek <chris.jerdonek at gmail.com> > wrote: >> On Thu, Jul 20, 2017 at 8:49 PM, Nick Coghlan <ncoghlan at gmail.com> >> wrote: >> > ... >> > * Lazy loading can have a significant impact on startup time, as it >> > means you don't have to pay for the cost of finding and loading >> > modules that you don't actually end up using on that particular run > > It should be mentioned that I have started designing an API to make > using > lazy loading much easier in Python 3.7 (i.e. "calling a single function" > easier), but I still have to write the tests and such before I propose a > patch and it will still be mainly for apps that know what they are doing > since lazy loading makes debugging import errors harder. > ... >> > However, if we're going to recommend them as good practices for 3rd >> > party developers looking to optimise the startup time of their Python >> > applications, then it makes sense for us to embrace them for the >> > standard library as well, rather than having our first reaction be to >> > write more hand-crafted C code. >> >> Are there any good write-ups of best practices and techniques in this >> area for applications (other than obvious things like avoiding >> unnecessary imports)? I'm thinking of things like how to structure >> your project, things to look for, developer tools that might help, and >> perhaps third-party runtime libraries? > > Nothing beyond "profile your application" and "don't do stuff during > import > as a side-effect" that I'm aware of. One "project structure" idea of the sort I had in mind is to move frequently used functions in a module into their own module. This way the most common paths of execution don't load unneeded functions. Following this line of reasoning could lead to grouping functions in an application by when they're needed instead of by what they do, which is different from what we normally see. I don't recall seeing advice like this anywhere, so maybe the trade-offs aren't worth it. Thoughts? --Chris

> > -Brett > >> >> >> --Chris >> >> >> >> > >> > On that last point, it's also worth keeping in mind that we have a >> > much harder time finding new C-level contributors than we do new >> > Python-level ones, and have every reason to expect that problem to >> > get >> > worse over time rather than better (since writing and maintaining >> > handcrafted C code is likely to go the way of writing and maintaining >> > handcrafted assembly code as a skillset: while it will still be >> > genuinely necessary in some contexts, it will also be an increasingly >> > niche technical specialty). >> > >> > Starting to migrate to using Cython for our acceleration modules >> > instead of plain C should thus prove to be a win for everyone: >> > >> > - Cython structurally avoids a lot of typical bugs that arise in >> > hand-coded extensions (e.g. refcount bugs) >> > - by design, it's much easier to mentally switch between Python & >> > Cython than it is between Python & C >> > - Cython accelerated modules are easier to adapt to other interpeter >> > implementations than handcrafted C modules >> > - keeping Python modules and their C accelerated counterparts in sync >> > will be easier, as they'll mostly be using the same code >> > - we'd be able to start writing C API test cases in Cython rather >> > than >> > in handcrafted C (which currently mostly translates to only testing >> > them indirectly) >> > - CPython's own test suite would naturally help test Cython >> > compatibility with any C API updates >> > - we'd have an inherent incentive to help enhance Cython to take >> > advantage of new C API features >> > >> > The are some genuine downsides in increasing the complexity of >> > bootstrapping CPython when all you're starting with is a VCS clone >> > and >> > a C compiler, but those complications are ultimately no worse than >> > those we already have with Argument Clinic, and hence amenable to the >> > same solution: if we need to, we can check in the generated C files >> > in >> > order to make bootstrapping easier. >> > >> > Cheers, >> > Nick. >> > >> > -- >> > Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia _>> > ________________________ >> > Python-Dev mailing list >> > Python-Dev at python.org >> > https://mail.python.org/mailman/listinfo/python-dev >> > Unsubscribe: >> > >> > https://mail.python.org/mailman/options/python-dev/chris.jerdonek%40gmail.com _>> ________________________ >> Python-Dev mailing list >> Python-Dev at python.org >> https://mail.python.org/mailman/listinfo/python-dev >> Unsubscribe: >> https://mail.python.org/mailman/options/python-dev/brett%40python.org On Fri, Jul 21, 2017 at 9:52 AM, Brett Cannon <brett at python.org> wrote: > > > On Thu, 20 Jul 2017 at 22:11 Chris Jerdonek <chris.jerdonek at gmail.com> > wrote: >> >> On Thu, Jul 20, 2017 at 8:49 PM, Nick Coghlan <ncoghlan at gmail.com> >> wrote: >> > ... >> > * Lazy loading can have a significant impact on startup time, as it >> > means you don't have to pay for the cost of finding and loading >> > modules that you don't actually end up using on that particular run > > > It should be mentioned that I have started designing an API to make > using > lazy loading much easier in Python 3.7 (i.e. "calling a single function" > easier), but I still have to write the tests and such before I propose a > patch and it will still be mainly for apps that know what they are doing > since lazy loading makes debugging import errors harder. > >> >> > >> > We've historically resisted adopting these techniques for the >> > standard >> > library because they do make things more complicated and harder >> > to >> > debug relative to plain old eagerly imported dynamic Python code. >> > However, if we're going to recommend them as good practices for 3rd >> > party developers looking to optimise the startup time of their Python >> > applications, then it makes sense for us to embrace them for the >> > standard library as well, rather than having our first reaction be to >> > write more hand-crafted C code. >> >> Are there any good write-ups of best practices and techniques in this >> area for applications (other than obvious things like avoiding >> unnecessary imports)? I'm thinking of things like how to structure >> your project, things to look for, developer tools that might help, and >> perhaps third-party runtime libraries? > > > Nothing beyond "profile your application" and "don't do stuff during > import > as a side-effect" that I'm aware of. > > -Brett > >> >> >> --Chris >> >> >> >> > >> > On that last point, it's also worth keeping in mind that we have a >> > much harder time finding new C-level contributors than we do new >> > Python-level ones, and have every reason to expect that problem to >> > get >> > worse over time rather than better (since writing and maintaining >> > handcrafted C code is likely to go the way of writing and maintaining >> > handcrafted assembly code as a skillset: while it will still be >> > genuinely necessary in some contexts, it will also be an increasingly >> > niche technical specialty). >> > >> > Starting to migrate to using Cython for our acceleration modules >> > instead of plain C should thus prove to be a win for everyone: >> > >> > - Cython structurally avoids a lot of typical bugs that arise in >> > hand-coded extensions (e.g. refcount bugs) >> > - by design, it's much easier to mentally switch between Python & >> > Cython than it is between Python & C >> > - Cython accelerated modules are easier to adapt to other interpeter >> > implementations than handcrafted C modules >> > - keeping Python modules and their C accelerated counterparts in sync >> > will be easier, as they'll mostly be using the same code >> > - we'd be able to start writing C API test cases in Cython rather >> > than >> > in handcrafted C (which currently mostly translates to only testing >> > them indirectly) >> > - CPython's own test suite would naturally help test Cython >> > compatibility with any C API updates >> > - we'd have an inherent incentive to help enhance Cython to take >> > advantage of new C API features >> > >> > The are some genuine downsides in increasing the complexity of >> > bootstrapping CPython when all you're starting with is a VCS clone >> > and >> > a C compiler, but those complications are ultimately no worse than >> > those we already have with Argument Clinic, and hence amenable to the >> > same solution: if we need to, we can check in the generated C files >> > in >> > order to make bootstrapping easier. >> > >> > Cheers, >> > Nick. >> > >> > -- >> > Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia _>> > ________________________ >> > Python-Dev mailing list >> > Python-Dev at python.org >> > https://mail.python.org/mailman/listinfo/python-dev >> > Unsubscribe: >> > >> > https://mail.python.org/mailman/options/python-dev/chris.jerdonek%40gmail.com _>> ________________________ >> Python-Dev mailing list >> Python-Dev at python.org >> https://mail.python.org/mailman/listinfo/python-dev >> Unsubscribe: >> https://mail.python.org/mailman/options/python-dev/brett%40python.org

Python-Dev mailing list Python-Dev at python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/victor.stinner%40gmail.com

More information about the Python-Dev mailing list