[Python-Dev] extremely slow exit for program having huge (45G) dict (python 2.5.2) (original) (raw)

Tim Peters tim.peters at gmail.com
Sat Dec 20 22:03:11 CET 2008

[Mike Coleman]

... Regarding interning, I thought this only worked with strings.

Implementation details. Recent versions of CPython also, e.g., "intern" the empty tuple, and very small integers.

Is there some way to intern integers? I'm probably creating 300M integers more or less uniformly distributed across range(10000)?

Interning would /vastly/ reduce memory use for ints in that case, from gigabytes down to less than half a megabyte.

[Scott David Daniels]

held = list(range(10000)) ... troublesomedict[string] = held[numbertohold] ...

More generally, but a bit slower, for objects usable as dict keys, change code of the form:

x = whatever_you_do_to_get_a_new_object()
use(x)

to:

x = whatever_you_do_to_get_a_new_object()
x = intern_it(x, x)
use(x)

where intern_it is defined like so once at the start of the program:

intern_it = {}.setdefault

This snippet may make the mechanism clearer:

internit = {}.setdefault x = 3000 id(internit(x, x)) 36166156 x = 1000 + 2000 id(internit(x, x)) 36166156 x = "works for computed strings too" id(internit(x, x)) 27062696 x = "works for computed strings t" + "o" * 2 id(internit(x, x)) 27062696

More information about the Python-Dev mailing list