Issue 1472695: 32/64bit pickled Random incompatiblity (original) (raw)

Issue1472695

Created on 2006-04-19 01:10 by pm67nz, last changed 2022-04-11 14:56 by admin. This issue is now closed.

Messages (5)
msg28294 - (view)	Author: Peter Maxwell (pm67nz)	Date: 2006-04-19 01:10
The unsigned long integers which make up the state of a Random instance are converted to Python integers via a cast to long in _randommodule.c's random_getstate function, so on a 32bit platform Random.getstate() returns a mix of postitive and negative integers, while on a 64bit platform the negative numbers are replaced by larger positive numbers, their 32bit-2s-complement equivalents. As a result, unpicking a Random instance from a 64bit machine on a 32bit platform produces the error "OverflowError: long int too large to convert to int". Unpickling a 32bit Random on a 64bit machine succeeds, but the resulting object is in a slightly confused state: >>> r32 = cPickle.load(open('r32_3.pickle')) >>> for i in range(3): ... print r64.random(), r32.random() ... 0.237964627092 4292886520.32 0.544229225296 0.544229225296 0.369955166548 4292886520.19
msg28295 - (view)	Author: Neal Norwitz (nnorwitz) *	Date: 2006-04-19 07:02
Logged In: YES user_id=33168 Peter, thanks for the report. Do you think you could work up a patch to correct this problem?
msg28296 - (view)	Author: Peter Maxwell (pm67nz)	Date: 2006-04-21 06:03
Logged In: YES user_id=320286 OK, here is a candidate patch, though I don't know if it is the best way to do it or meets the style guidelines etc. It makes Random pickles interchangable between 32bit and 64bit machines by encoding their states as Python long integers. An old pre-patch 32bit pickle loaded on a 64bit machine still fails (OverflowError: can't convert negative value to unsigned long) but I hope that combination is rare enough to ignore. Also on a 32bit machine new Random pickles can't be unpickled by a pre-patch python, but again there are limits to sane backward compatability.
msg28297 - (view)	Author: Raymond Hettinger (rhettinger) *	Date: 2006-04-25 23:00
Logged In: YES user_id=80475 Tim, do you think we should require that the world not change for 32-bit pickles?
msg28298 - (view)	Author: Tim Peters (tim.peters) *	Date: 2006-04-26 00:26
Logged In: YES user_id=31435 > do you think we should require that the world not > change for 32-bit pickles? I don't understand the question. If a pre-2.5 pickle here can be read in 2.5, where both producer & consumer are the same 32-vs-64 bit choice; and a 2.5+ pickle here is portable between 32- and 64- boxes, I'd say "good enough". While desirable, it's not really critical that a 2.5 pickle here be readable by an older Python. While that's critical for pickle in general, and critical too for everyone-uses-'em types (ints, strings, lists, ...), when fixing a bug in a specific rarely-used type's pickling strategy some slop is OK. IOW, it's just not worth heroic efforts to hide all pain. The docs should mention incompatibilities, though. Does that answer the question?