[Python-Dev] reference counting in Py3K (original) (raw)

Greg Ewing greg.ewing at canterbury.ac.nz
Thu Sep 8 06:31:17 CEST 2005

Phillip J. Eby wrote:

Just an FYI; Pyrex certainly makes it relatively painless to write code that interfaces with C, but it doesn't do much for performance, and naively-written Pyrex code can actually be slower than carefully-optimized Python code. So, for existing modules that were written in C for performance reasons, Pyrex isn't currently a substitute.

If the performance-critical parts of the C code were translated into Pyrex-C (i.e. Pyrex code that only performs C operations) it shouldn't be significantly different. Alternatively, Pyrex could be used to create a wrapper around the existing C code, freeing it from having to deal with the Python/C API, thus making it easier to maintain.

One of the reasons for this is that Pyrex code uses the generic Python/C APIs, like PySequenceGetItem, even in cases where PyListGetItem or its macro form would be more appropriate. Pyrex has no way currently to say, "this is type X's C API, so use it when you have a variable that's of type X, instead of using the generic object protocols."

I'm thinking of teaching it about some of the builtin types (e.g. list, dict) so it can use more efficient APIs when appropriate.

In the meantime, you can declare and call most of the APIs explicitly if you want. The exception is some of the macro forms which have nonstandard refcount behaviour. I'm also thinking of adding some declarations to help with that.

There are other issues that contribute to the inefficiency as well, like redundant refcounting, assigning None to temporary variables, etc.

Yes, it could probably do with a bit of flow analysis and peephole optimisation to deal with things like that.

I haven't used the absolute latest version of Pyrex, but older versions also used C strings for attribute lookups, which was horribly slow. I think the latest version now creates string objects at module initialization to avoid this issue, though.

Yes, it now precreates and interns all identifier strings, which should help considerably.

-- Greg Ewing, Computer Science Dept, +--------------------------------------+ University of Canterbury, | A citizen of NewZealandCorp, a | Christchurch, New Zealand | wholly-owned subsidiary of USA Inc. | greg.ewing at canterbury.ac.nz +--------------------------------------+

More information about the Python-Dev mailing list