[Python-Dev] Re: Another test_compiler mystery (original) (raw)

Michael Hudson mwh at python.net
Tue Aug 10 21:00:14 CEST 2004

Tim Peters <tim.peters at gmail.com> writes:

Well, this gets nasty.

You're certainly not wrong about that!

[snip Windows details]

That's

if (PyEnterRecursiveCall(" in cmp")) return NULL; in PyObjectRichCompare. That's just trying to compare two ints. So NULL gets returned to PyObjectRichCompareBool, which in turn returns -1 to lookdict. AAAARGHGH! lookdict "isn't allowed" to raise exceptions, so it does a PyErrClear(), goes on to futilely chase the entire dict looking for another match on the hash code, and we've effectively turned a MemoryError into a KeyError. I expect that explains a lot about what we see in the release-build runs.

This part happens on Linux, too, it seems. Running your script:

114 ok 115 ok 116 ok 117 ok 118 ok 119 ok 120 ok 121 ok 122 ok 123 exceptions.RuntimeError maximum recursion depth exceeded 124 exceptions.RuntimeError maximum recursion depth exceeded 125 exceptions.RuntimeError maximum recursion depth exceeded 126 exceptions.KeyError 307 127 exceptions.RuntimeError maximum recursion depth exceeded 128 exceptions.KeyError 306 129 exceptions.RuntimeError maximum recursion depth exceeded 130 exceptions.KeyError 305 131 exceptions.RuntimeError maximum recursion depth exceeded 132 exceptions.KeyError 304 133 exceptions.RuntimeError maximum recursion depth exceeded 134 exceptions.KeyError 303 135 exceptions.KeyError 302 136 exceptions.RuntimeError maximum recursion depth exceeded 137 exceptions.KeyError 301

This is a debug build, so we're not abusing the stack quite so much on linux as apparently seems to happen in debug builds on windows.

This isn't going to be something we can pin on the new Windows tools, is it? It doesn't seem like a feature that a debug build gets a tiny stack, but, well...

The question is what we did since 2.3.4 that apparently increases our stack demands, and grossly increases them in a debug build(!). ^ on windows; I don't see anything so scary on Linux.

Hmm, maybe that's a lie, I don't see the crazy KeyErrors with 2.3.

Could be that the compile package is more heavily recursive now too (no idea).

Really don't think so.

testparser.py in 2,3.4 contained the same deeply nested tuples, so that's not what changed.

Something that has changed, however, is that we lost the code that compared objects recursively and now PyObject_RichCompare uses the more general Py_EnterRecursiveCall mechanism and not its own counter. So now we're grossly more likely to hit one of these faux KeyErrors in 2.3 than 2.4 (you'd need to invoke a Python eq method or so).

Would it make more sense for PyObject_RichCompare to still use a private counter but raise RuntimeError if it is exceeded instead of trying to be clever? That would make this problem /less/ likely, though obviously still possible (functions that claim they 'can't fail' are lying).

Running Misc/find_recursionlimit.py with a debug build yields ... well, I gave up and killed it at 22000 (hmm, seems the default 'ulimit -s' is "unlimited"... I'm sure this hasn't always had this effect with a threaded build).

Cheers, mwh

--

Emacs is a fashion statement. No, Gnus is a fashion statement. Emacs is clothing. Everyone else is running around naked. -- Karl Kleinpaste & Jonadab the Unsightly One, gnu.emacs.gnus

More information about the Python-Dev mailing list