[Python-Dev] Split unicodeobject.c into subfiles (original) (raw)

Victor Stinner victor.stinner at gmail.com
Tue Oct 23 12:05:21 CEST 2012

Such a restructuring should not result in compilers no longer being able to optimize code by inlining functions in one of the most important basic types we have in Python 3.

I agree that performances are important. But I'm not convinced than moving functions has a real impact on performances, not that such issues cannot be fixed.

I tried to limit changes impacting performances. Inlining is (only?) interesting for short functions. PEP 393 introduces many macros for this. I also added some "Fast" functiions (_PyUnicode_FastCopyCharacters() and _PyUnicode_FastFill()) which don't check parameters and do the real work. I don't think that it's really useful to inline _PyUnicode_FastFill() in the caller for example.

I will check performances of all str methods. For example, str.count() is now calling PyUnicode_Count() instead of the static count(). PyUnicode_Count() adds some extra checks, some of them are not necessary, and it's not a static function, so it cannot(?) be inlined. But I bet that the overhead is really low.

Note: Since GCC 4.5, Link Time Optimization are possible. I don't know if GCC is able to inline functions defined in different files, but C compilers are better at each release.

--

I will check the impact of performances on _PyUnicode_Widen() and _PyUnicode_Putchar(), which are no more static. _PyUnicode_Widen() and _PyUnicode_Putchar() are used in Unicode codecs when it's more expensive to compute the exact length and maximum character of the output string. These functions are optimistic (hope that the output will not grow too much and the string is not "widen" too much times, so it should be faster for ASCII).

I implemented a similar approach in my PyUnicodeWriter API, and I plan to reuse this API to simplify the API. PyUnicodeWriter uses some macro to limit the overhead of having to check before each write if we need to enlarge or widen the internal buffer, and allow to write directly into the buffer using low level functions like PyUnicode_WRITE.

I also hope a performance improvement because the PyUnicodeWriter API can also overallocate the internal buffer to reduce the number of calls to realloc() (which is usually slow).

Also note that splitting the file in multiple smaller ones will actually create more maintenance overhead, since patches will likely no longer be easy to merge from 3.3 to 3.4.

I'm a candidate to maintain unicodeobject.c. In your check unicodeobject.c (recent) history, I'm one of the most active developer on this file since two years (especially in 2012). I'm not sure that merges on this file are so hard.

Victor

More information about the Python-Dev mailing list