[Python-Dev] PEP 393 review (original) (raw)

"Martin v. Löwis" martin at v.loewis.de
Thu Aug 25 10:24:39 CEST 2011

With this PEP, the unicode object overhead grows to 10 pointer-sized words (including PyObjectHEAD), that's 80 bytes on a 64-bit machine. Does it have any adverse effects?

If I count correctly, it's only three additional words (compared to 3.2): four new ones, minus one that is removed. In addition, it drops a memory block. Assuming a malloc overhead of two pointers per malloc block, we get one additional pointer.

On a 32-bit machine with a 32-bit wchar_t, pure-ASCII strings of length 1 (+NUL) will take the same memory either way: 8 bytes for the characters in 3.2, 2 bytes in 3.3 + extra pointer + padding. Strings of 2 or more characters will take more space in 3.2.

On a 32-bit machine with a 16-bit wchar_t, pure-ASCII strings up to 3 characters take the same space either way; space savings start at four characters.

On a 64-bit machine with a 16-bit wchar_t, assuming a malloc minimum block size of 16 bytes, pure-ASCII strings of up to 7 characters take the same space. For 8 characters, 3.2 will need 32 bytes for the characters, whereas 3.3 will only take 16 bytes (due to padding).

So: no, I can't see any adverse effects. Details depend on the malloc implementation, though. A slight memory increase may occur on compared to a narrow build may occur for strings that use non-Latin-1, and a large increase for strings that use non-BMP characters.

The real issue of memory consumption is the alternative representations, if created. That applies for the default encoding in 3.2 as well as the wchar_t and UTF-8 representations in 3.3.

Are there any plans to make instantiation of small strings fast enough? Or is it already as fast as it should be?

I don't have any plans, and I don't see potential. Compared to 3.2, it saves a malloc call, which may be quite an improvement. OTOH, it needs to iterate over the characters twice, to find the largest character.

If you are referring to the reuse of Unicode objects: that's currently not done, and is difficult to do in the 3.2 way due to the various sizes of characters. One idea might be to only reuse UCS1 strings, and then keep a freelist for these based on the string length.

When interfacing with the Win32 "wide" APIs, what is the recommended way to get the required LPCWSTR?

As before: PyUnicode_AsUnicode.

Will the format codes returning a PyUNICODE pointer with PyArgParseTuple be deprecated?

Not for 3.3, no.

Do you think the wstr representation could be removed in some future version of Python?

Yes. This probably has to wait for Python 4, though.

Is PyUnicodeReady() necessary for all unicode objects, or only those allocated through the legacy API?

Only for the latter (although it doesn't hurt to apply it to all of them).

“The PyUnicode representation is not instantaneously available”: you mean the PyUNICODE representation?

Thanks, fixed.

- conditions you would like to pose on the implementation before acceptance. I'll see which of these can be resolved, and list the ones that remain open. That it doesn't significantly slow down benchmarks such as stringbench and iobench.

Can you please quantify "significantly"? Also, having a complete list of benchmarks to perform prior to acceptance would be helpful.

Thanks, Martin

More information about the Python-Dev mailing list