[Python-Dev] Impact of Namedtuple on startup time (original) (raw)

Antoine Pitrou solipsis at pitrou.net
Tue Jul 18 08:28:51 EDT 2017

On Tue, 18 Jul 2017 22:13:01 +1000 Nick Coghlan <ncoghlan at gmail.com> wrote:

As another example of this: while trading the global import lock for per-module locks eliminated most of the old import deadlocks, it turns out that it also left us with some fairly messy race conditions and more fragile code (I still count that particular case as a win overall, but it definitely raises the barrier to entry for maintaining that code). Unfortunately, these are frequently cases where the benefits are immediately visible (e.g. faster benchmark results, removing longstanding limitations on user code), but the downsides can literally take years to make themselves felt (e.g. higher defect rates in the interpreter, subtle bugs in previously correct user code that are eventually traced back to interpreter changes).

The import deadlocks were really in the category of "subtle bugs" that only occur in certain timing conditions (especially when combined with PyImport_ImportModuleNoBlock and/or stdlib modules which can try to import stuff silently, such as the codecs module). So we traded a category of "subtle bugs" due to a core design deficiency for another category of "subtle bugs" due to an imperfect implementation, the latter being actually fixable incrementally :-)

Disclaimer: I wrote the initial per-module lock implementation, which was motivated by those long-standing "subtle bugs" in multi-threaded applications.

Regards

Antoine.

More information about the Python-Dev mailing list