[Python-Dev] Issue 14417: consequences of new dict runtime error (original) (raw)

Guido van Rossum guido at python.org
Thu Mar 29 22:09:17 CEST 2012

On Thu, Mar 29, 2012 at 12:58 PM, R. David Murray <rdmurray at bitdance.com> wrote:

Some of us have expressed uneasiness about the consequences of dict raising an error on lookup if the dict has been modified, the fix Victor made to solve one of the crashers.

I don't know if I speak for the others, but (assuming that I understand the change correctly) my concern is that there is probably a significant amount of threading code out there that assumes that dict lookup is a thread-safe operation.  Much of that code will, if moved to Python 3.3, now be subject to random runtime errors for which it will not be prepared.  Further, code which appears safe can suddenly become unsafe if a refactoring of the code causes an object to be stored in the dictionary that has a Python equality method.

My original assessment was that this only affects dicts whose keys have a user-implemented hash or eq implementation, and that the number of apps that use this and assume the threadsafe property would be pretty small. This is just intuition, I don't have hard facts. But I do want to stress that not all dict lookups automatically become thread-unsafe, only those that need to run user code as part of the key lookup.

Would it be possible to modify the fix so that the lookup is retried a non-trivial but finite number of times, so that normal code will work and only pathological code will break?

FWIW a similar approach was rejected as a fix for the hash DoS attack.

I know that I really don't want to think about having to audit the (significantly threaded) application I'm currently working on to make sure it is "3.3 safe".  Dict lookup operations are common, and we've never had to think about whether or not they were thread-safe before (unless there were inter-thread synchronization issues involved, of course). Nor am I sure the locking dict type suggested by Jim on the issue would help, since a number of the dicts we are using are produced by library code.  So we'd have to wait for those libraries to be ported to 3.3....

Agreed that this is somewhat scary.

-- --Guido van Rossum (python.org/~guido)

More information about the Python-Dev mailing list