[Python-Dev] FW: METH_COEXIST (original) (raw)

Raymond Hettinger python at rcn.com
Fri Dec 12 10:55:04 EST 2003

Long ago, I noticed that non-operator calls to dict.contains() ran about 50% slower than dict.has_key(). Since then, the issue has surfaced several times for other wrapped methods like init(), getitem(), and setitem(), and next(). Also, it came up again in the context of set.contains. The latter bugged me enough to get to the bottom of it (unlike dicts, sets do not have has_key() as an alternative so you feel it when writing ismember=myset.contains).

Over 95% of the speed difference is due to PyCFunctions having optimized calls. Of that, much of the benefit comes from the METH_O and METH_NOARGS methods having their arguments passed directly instead of by assembling and disassembling an argument tuple.

If the problem were widespread and important in every context, the solution would be to provide wrapperobjects with special cases just like PyCFunction.

Instead, there is a minimally invasive alternative. If I could just write a "contains" method as a PyCFunction, the job would be done. Unfortunately, if the corresponding slot has already filled-in the method with a wrapper object, the explicitly defined method is ignored. While that is normally fine, it is possible to make the explicitly defined method coexist with the slot.

So, I propose to add a method flag, METH_COEXIST, which allows an entry in the method table to overwrite the wrapperobject and coexist with the slot:

*** typeobject.c 13 Nov 2003 22:50:00 -0000 2.253 --- typeobject.c 12 Dec 2003 13:58:42 -0000

*** 2792,2799 ****

    for (; meth->ml_name != NULL; meth++) {
            PyObject *descr;

! if (PyDict_GetItemString(dict, meth->ml_name)) ! continue; if (meth->ml_flags & METH_CLASS) { if (meth->ml_flags & METH_STATIC) { PyErr_SetString(PyExc_ValueError, --- 2792,2802 ----

    for (; meth->ml_name != NULL; meth++) {
            PyObject *descr;

! if (PyDict_GetItemString(dict, meth->ml_name)) { ! if (!(meth->ml_flags & METH_COEXIST)) ! continue; ! PyDict_DelItemString(dict, meth->ml_name); ! } if (meth->ml_flags & METH_CLASS) { if (meth->ml_flags & METH_STATIC) { PyErr_SetString(PyExc_ValueError,

With that minor change, it becomes possible to write:

static PyMethodDef mapp_methods[] = { {"contains", (PyCFunction)dict_has_key, METH_O | METH_COEXIST, contains__doc__},

And, presto, now calls to dict.contains() are as fast as dict.has_key().

The two immediate applications are dict.contains() and set.contains().

Guido is basically okay with the idea but thought I should check here first to see if anyone has any profound thoughts about unexpected implications or perhaps a better way.

If not, then I would like to make the above changes (define the flag, teach typeobject.c to recognize the flag, and apply it to dicts and sets).

Raymond Hettinger

More information about the Python-Dev mailing list