[Python-Dev] Cython compiled stdlib modules (original) (raw)

Nick Coghlan ncoghlan at gmail.com
Mon Jul 24 10:25:04 EDT 2017

On 22 July 2017 at 06:43, Stefan Behnel <stefan_ml at behnel.de> wrote:

Nick Coghlan schrieb am 21.07.2017 um 08:23:

I'll also note that in these cases where the import overhead is proportionally significant for always-imported modules, we may want to look at the benefits of freezing them (if they otherwise remain as pure Python modules), or compiling them as builtin modules (if we switch them over to Cython), in addition to looking at ways to make the modules themselves faster. Just for the sake of it, I gave the Cython compilation a try. I had to apply the attached hack to Lib/typing.py to get the test passing, because it uses frame call offsets in some places and Cython functions do not create frames when being called (they only create them for exception traces). I also had to disable the import of "abc" in the Cython generated module to remove the circular self dependency at startup when the "abc" module is compiled. That shouldn't have an impact on the runtime performance, though.

[snip]

As it stands, the gain is probably not worth the increase in library file size, which also translates to a higher bottom line for the memory consumption. At least not for these two modules. Manually optimising the files would likely also reduce the .so file size in addition to giving better speedups, though, because the generated code would become less generic.

Thanks for trying the experiment! I agree with your conclusion that the file size impact likely rules it out as a general technique.

Selective freezing may still be interesting though, since that at least avoids the import path searches and merges the disk read into the initial loading of the executable.

Cheers, Nick.

-- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia

More information about the Python-Dev mailing list