(original) (raw)
On Sat, 3 Sep 2016 at 16:43 Yury Selivanov <yselivanov.ml@gmail.com> wrote:
On 2016-09-03 4:15 PM, Christian Heimes wrote:
\> On 2016-09-04 00:03, Yury Selivanov wrote:
\>>
\>> On 2016-09-03 12:27 PM, Brett Cannon wrote:
\>>> Below is the \`co\_extra\` section of PEP 523 with the update saying that
\>>> users are expected to put a tuple in the field for easier simultaneous
\>>> use of the field.
\>>>
\>>> Since the \`co\_extra\` discussions do not affect CPython itself I'm
\>>> planning on landing the changes stemming from the PEP probably on Monday.
\>> Tuples are immutable. If you have multiple co\_extra users then they
\>> will have to either mutate tuple (which isn't always possible, for
\>> instance, you can't increase size), or to replace it with another tuple.
\>>
\>> Creating lists is a bit more expensive, but item access speed should be
\>> in the same ballpark.
\>>
\>> Another question -- sorry if this was discussed before -- why do we want
\>> a PyObject\* there at all? I.e. why don't we create a dedicated struct
\>> CoExtraContainer to manage the stuff in co\_extra? My understanding is
\>> that the users of co\_extra are C-level python optimizers and profilers,
\>> which don't need the overhead of CPython API.
As Chris pointed out in another email, the overhead is only in the allocation, not the iteration/access if you use the PyTuple macros to get the size and index into the tuple the overhead is negligible.
\>>
\>> This way my work to add an extra caching layer (which I'm very much
\>> willing to continue to work on) wouldn't require another set of extra
\>> fields for code objects.
\> Quick idea before I go to bed:
\>
\> You could adopt a similar API to OpenSSL's CRYPTO\_get\_ex\_new\_index()
\> API,
\> https://www.openssl.org/docs/manmaster/crypto/CRYPTO\_get\_ex\_new\_index.html
\>
\>
\> static int code\_index = 0;
\>
\> int PyCodeObject\_NewIndex() {
\> return code\_index++;
\> }
\>
\> A library like Pyjion has to acquire an index first. In further calls it
\> uses the index as offset into the new co\_extra field. Libraries don't
\> have to hard-code their offset and two libraries will never conflict.
\> PyCode\_New() can pre-populate co\_extra with a PyTuple of size
\> code\_index. This avoids most resizes if you load Pyjion early. For
\> code\_index == 0 leaf the field NULL.
Sounds like a very good idea!
The problem with this is the pre-population. If you don't get your index assigned before the very first code object is allocated then you still have to manage the size of the tuple in co\_extra. So what this would do is avoid the iteration but not the allocation overhead.
If we open up the can of worms in terms of custom functions for this (which I was trying to avoid), then you end up with Py\_ssize\_t \_PyCode\_ExtraIndex(), PyObject \*
\_PyCode\_GetExtra(PyCodeObject \*code, Py\_ssize\_t index), and int \_PyCode\_SetExtra(PyCodeObject \*code, Py\_ssize\_t index, PyObject \*data) which does all the right things for creating or resizing the tuple as necessary and which I think matches mostly what Nick had proposed earlier. But the pseudo-code for \_PyCode\_GetExtra() would be::
if co\_extra is None:
co\_extra = (None,) \* \_next\_extra\_index;
return None
elif len(co\_extra) < index - 1:
... pad out tuple
return None
else:
return co\_extra\[index\]
Is that going to save us enough to want to have a custom API for this?
-Brett
Yury