[Python-Dev] new buffer in python2.7 (original) (raw)

Kristján Valur Jónsson kristjan at ccpgames.com
Wed Oct 27 16:28:16 CEST 2010

So, I did some profiling. It turns out that the difference is not due to the creation of the MemoryView object as such, but the much more complicated slice handling for generic objects. Here is the interesting part of the inclusive sample measurements for the MemoryView: Function Name Inclusive Samples Exclusive Samples Inclusive Samples % Exclusive Samples % apply_slice 8.997 468 62,23 3,24 _PyObject_GetItem 6.257 400 43,28 2,77 memory_subscript 5.857 1.051 40,51 7,27 _PyMemoryView_FromBuffer 2.642 455 18,27 3,15 memory_dealloc 1.572 297 10,87 2,05 _PyObject_Malloc 1.374 1.374 9,50 9,50 __PyObject_GC_New 1.256 236 8,69 1,63 _PySlice_New 1.211 333 8,38 2,30 slice_dealloc 1.061 769 7,34 5,32 __PyObject_GC_Malloc 1.022 293 7,07 2,03 bytearray_getbuffer 987 354 6,83 2,45 dup_buffer 932 932 6,45 6,45

Notice how a Slice object is generated. Then a PyObject_GetItem() is done. The salient code path is from apply_slice(). A slice object must be constructed and destroyed.

In contrast, when done on a string directly (or a bytes object in py3k) you go directly to PySequence_GetSlice. Function Name Inclusive Samples Exclusive Samples Inclusive Samples % Exclusive Samples % apply_slice 3.888 502 48,44 6,25 _PySequence_GetSlice 3.039 350 37,86 4,36 string_slice 2.689 281 33,50 3,50 _PyString_FromStringAndSize 2.409 575 30,01 7,16 [MSVCR90.dll] 1.413 1.407 17,61 17,53 string_dealloc 467 150 5,82 1,87 _PyObject_Malloc 379 379 4,72 4,72 _PyObject_Free 378 378 4,71 4,71 __PyEval_SliceIndex 347 347 4,32 4,32

(The measurements for the MemoryView above already contain some optimizations I've done on naïve functions). This area is ripe for improvements.

K -----Original Message----- From: python-dev-bounces+kristjan=ccpgames.com at python.org [mailto:python-dev-bounces+kristjan=ccpgames.com at python.org] On Behalf Of Kristján Valur Jónsson Sent: Wednesday, October 27, 2010 20:00 To: Antoine Pitrou; python-dev at python.org Subject: Re: [Python-Dev] new buffer in python2.7

Ah, well in 2.7 you don't have the luxury of a bytes object. A str() would be similar in 2.7 Anyway, isn't the "bytes" object immutable? In that case, it is not a useful target for a sock.recv_into() call. Calling getbuffer on a bytearray or a bytes object should be really cheap, so I still don't accept this as a matter of fact situation.

Btw, going to 3.2 isn't a real option for us any time soon. A lot of companies probably find themselves in a similar situation. This is why I spend so much effort applying some love to 2.7. Most of the stuff I locally do to 2.7 I then contribute to python as 3.2 patches. Someday they'll get backported, no doubt :)

K

-----Original Message----- From: python-dev-bounces+kristjan=ccpgames.com at python.org [mailto:python-dev-bounces+kristjan=ccpgames.com at python.org] On Behalf Of Antoine Pitrou Sent: Wednesday, October 27, 2010 19:16 To: python-dev at python.org Subject: Re: [Python-Dev] new buffer in python2.7

>Here are micro-benchmarks under 3.2:

> $ ./python -m timeit -s "x = b'x'*10000" "x[:100]" > 10000000 loops, best of 3: 0.134 usec per loop $ ./python -m timeit > -s "x = memoryview(b'x'*10000)" "x[:100]" > 10000000 loops, best of 3: 0.151 usec per loop That's weird. The greedy slice needs two memory allocations. One for the ByteArray object itself, one for its cargo. In total, more that 100 bytes. In contrast, creating the MemoryView object requires only one allocation of a few dozen bytes.

It's not a bytearray object, it's a bytes object. It requires only a single allocation since the data is allocated inline. The memoryview object must also call getbuffer() again on the original object.

Regards

Antoine.

Python-Dev mailing list Python-Dev at python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/kristjan%40ccpgames.com

Python-Dev mailing list Python-Dev at python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/kristjan%40ccpgames.com

More information about the Python-Dev mailing list