| msg31399 - (view) |
Author: Bartlomiej Ogryczak (oneg) |
Date: 2007-03-02 11:10 |
| Seems that on 64-bit machines cPickle is able to pickle floats smaller then DBL_MIN, but it's unable to unpickle them. 64-bit machine (Solaris 10 on Sparc) Python 2.4.4 (#1, Oct 26 2006, 10:01:37) [GCC 3.3.4] on sunos5 Type "help", "copyright", "credits" or "license" for more information. >>> import cPickle >>> cPickle.dumps(1e-310) 'F9.9999999999999694e-311\n.' >>> cPickle.loads(_) Traceback (most recent call last): File "", line 1, in ? ValueError: could not convert string to float Same thing works fine on 32-bit machine (P4 with MS Windows) Python 2.4.3 (#69, Mar 29 2006, 17:35:34) [MSC v.1310 32 bit (Intel)] on win32 Type "help", "copyright", "credits" or "license" for more information. >>> import cPickle >>> cPickle.dumps(1e-310) 'F0\n.' >>> |
|
|
| msg31400 - (view) |
Author: Santiago Gala (sgala) |
Date: 2007-03-05 16:38 |
| FYI: it doesn't happen for me in python 2.4.3, 2.5svn and 2.6svn built on linux (x86_64) with gcc-4.1.2, so it looks compiler version or hw platform specific. |
|
|
| msg31401 - (view) |
Author: Bartlomiej Ogryczak (oneg) |
Date: 2007-03-05 16:49 |
| I forgot to specify, it's 32-bit Python binary. I'm guessing yours is 64-bit binary on 64-bit system? Some additional info: gcc --version gcc (GCC) 3.3.4 $ file `which python` ELF 32-bit MSB executable SPARC Version 1, dynamically linked, not stripped $ ldd `which python` libresolv.so.2 => /usr/lib/libresolv.so.2 libsocket.so.1 => /usr/lib/libsocket.so.1 libnsl.so.1 => /usr/lib/libnsl.so.1 librt.so.1 => /usr/lib/librt.so.1 libdl.so.1 => /usr/lib/libdl.so.1 libstdc++.so.5 => /usr/local/lib/libstdc++.so.5 libm.so.2 => /usr/lib/libm.so.2 libgcc_s.so.1 => /usr/local/lib/libgcc_s.so.1 libc.so.1 => /usr/lib/libc.so.1 libmp.so.2 => /usr/lib/libmp.so.2 libmd5.so.1 => /usr/lib/libmd5.so.1 libscf.so.1 => /usr/lib/libscf.so.1 libaio.so.1 => /usr/lib/libaio.so.1 libdoor.so.1 => /usr/lib/libdoor.so.1 libuutil.so.1 => /usr/lib/libuutil.so.1 /platform/SUNW,Sun-Fire-15000/lib/libc_psr.so.1 /platform/SUNW,Sun-Fire-15000/lib/libmd5_psr.so.1 |
|
|
| msg31402 - (view) |
Author: Georg Brandl (georg.brandl) *  |
Date: 2007-03-05 21:46 |
| I seem to remember that float pickling has been overhauled for 2.5, could you try with a 2.5 Python and report the results? |
|
|
| msg31403 - (view) |
Author: Martin v. Löwis (loewis) * |
Date: 2007-03-10 17:09 |
| If python is a 32-bit binary, it should be completely irrelevant that the processor is a 64-bit processor. Python can't tell the difference. |
|
|
| msg79134 - (view) |
Author: Mark Dickinson (mark.dickinson) * |
Date: 2009-01-05 12:22 |
| I'm seeing this too on OS X, with Python 2.6. Python 2.6.1+ (release26-maint:68182M, Jan 2 2009, 23:13:43) [GCC 4.0.1 (Apple Inc. build 5490)] on darwin Type "help", "copyright", "credits" or "license" for more information. >>> import cPickle >>> cPickle.dumps(1e-307) 'F9.9999999999999991e-308\n.' >>> cPickle.loads(_) 9.9999999999999991e-308 >>> cPickle.dumps(1e-308) 'F9.9999999999999991e-309\n.' >>> cPickle.loads(_) Traceback (most recent call last): File "", line 1, in ValueError: could not convert string to float Same results on trunk, and (using pickle instead of cPickle, and specifying protocol 0 explicitly) for 3.x. I'll see if I can figure out where this is coming from. |
|
|
| msg79150 - (view) |
Author: Mark Dickinson (mark.dickinson) * |
Date: 2009-01-05 14:07 |
| The problem comes down to the platform strtod: on some systems, strtod sets errno to ERANGE on underflow. The load_float function in Modules/cPickle.c calls PyOS_ascii_strtod and then raises ValueError if that call sets errno. I suggest replacing the call to PyOS_ascii_strtod with a call to PyFloat_FromString instead. This would make cPickle behave identically to pickle, and would also fix the problem on Windows where 1e-310 is dumped as 0.0. (PyFloat_FromString goes out of its way to use atof instead of strtod to deal with this case.) |
|
|
| msg79154 - (view) |
Author: Mark Dickinson (mark.dickinson) * |
Date: 2009-01-05 15:18 |
| Change title; this has nothing to do with 64 bit. |
|
|
| msg79949 - (view) |
Author: Mark Dickinson (mark.dickinson) * |
Date: 2009-01-16 10:35 |
| Here's a patch, against the trunk, that simply amends load_float to ignore errno on underflow (while retaining it on overflow). This fixes the problem on my machine. The added tests may be a little severe. There's no *good* reason why dumping and then loading a float on the same machine shouldn't produce exactly the same value, but there are likely many bad reasons. I propose to checkin the change and watch the buildbots carefully; if there are failures I'll weaken the tests to check for approximate equality instead of exact equality. |
|
|
| msg80464 - (view) |
Author: Mark Dickinson (mark.dickinson) * |
Date: 2009-01-24 17:00 |
| Fixed in the trunk in r68903. If the buildbots are happy with the new tests, I'll port to 2.6, 3.0 and 3.1. |
|
|
| msg80472 - (view) |
Author: Mark Dickinson (mark.dickinson) * |
Date: 2009-01-24 19:07 |
| Hmm. After checking in this patch, the gentoo x86 trunk buildbot has a new failure in test_pickletools, with the following output: test_pickletools ********************************************************************** File "/home/buildslave/python-trunk/trunk.norwitz-x86/build/Lib/pickletools.py", line ?, in pickletools.__test__.disassembler_test Failed example: dis(pickle.dumps(random.random, 0)) Expected: 0: c GLOBAL 'random random' 15: p PUT 0 18: . STOP highest protocol among opcodes = 0 Got: 0: c GLOBAL 'bsddb.test.test_thread random' 31: p PUT 0 34: . STOP highest protocol among opcodes = 0 ********************************************************************** 1 items had failures: 1 of 25 in pickletools.__test__.disassembler_test ***Test Failed*** 1 failures. test test_pickletools failed -- 1 of 95 doctests failed I'm having some difficulty figuring out how this could be related to the checkin of this patch. Anyone have any ideas where this might be coming from? |
|
|
| msg80475 - (view) |
Author: Antoine Pitrou (pitrou) * |
Date: 2009-01-24 19:54 |
| This is not a new failure but an erratic one, and caused by a weakness in pickle's handling of functions. See #3657. |
|
|
| msg80476 - (view) |
Author: Mark Dickinson (mark.dickinson) * |
Date: 2009-01-24 19:59 |
| Thanks, Antoine! |
|
|
| msg80484 - (view) |
Author: Mark Dickinson (mark.dickinson) * |
Date: 2009-01-24 21:49 |
| Fix merged in r68907 (2.6), r68908 (3.1) and r68909 (3.0). |
|
|