| msg63523 - (view) |
Author: Trent Nelson (trent) *  |
Date: 2008-03-14 12:57 |
| S:\src\svn\svn.python.org\projects\python\trunk.x64\PCbuild>amd64\python_d ..\lib\test\test_marshal.py test_bool (__main__.IntTestCase) ... ok test_int64 (__main__.IntTestCase) ... ok test_ints (__main__.IntTestCase) ... ok test_floats (__main__.FloatTestCase) ... ok test_buffer (__main__.StringTestCase) ... ok test_string (__main__.StringTestCase) ... ok test_unicode (__main__.StringTestCase) ... ok test_code (__main__.CodeTestCase) ... ok test_dict (__main__.ContainerTestCase) ... ok test_list (__main__.ContainerTestCase) ... ok test_sets (__main__.ContainerTestCase) ... ok test_tuple (__main__.ContainerTestCase) ... ok test_exceptions (__main__.ExceptionTestCase) ... ok test_bug_5888452 (__main__.BugsTestCase) ... ok test_exact_type_match (__main__.BugsTestCase) ... ok test_fuzz (__main__.BugsTestCase) ... ok test_loads_recursion (__main__.BugsTestCase) ... ^^^ python_d.exe crashes at this point with a stack overflow. Just capturing the issue for now. Will investigate shortly and hopefully provide a patch. |
|
|
| msg63536 - (view) |
Author: Trent Nelson (trent) * |
Date: 2008-03-14 20:36 |
| Traced the problem down to the following minimal code snippet: import marshal s = 'c' + ('X' * 4*4) + '{' * 2**20 marshal.loads(s) When Python/marshal.c:18 MAX_MARSHAL_STACK_DEPTH is 2000 (which is what it is currently), marshal.loads() eventually overflows the stack in r_object(). There is a check in r_object() to avoid this though: if (p->depth > MAX_MARSHAL_STACK_DEPTH) { p->depth--; PyErr_SetString(PyExc_ValueError, "recursion limit exceeded"); return NULL; } On Windows x64, a value of 1964 raises the recursion limit exception above (which is what test_marshal is expecting). With a value of 1965, a C stack overflow exception is raised. So, MAX_MARSHAL_STACK_DEPTH needs to be <= 1964 in order to prevent this particular code from overflowing the stack on Win64 before we can raise a Python recursion limit exception. Was there any science behind choosing 2000 as the current value? Should a new value (e.g. 1500) be provided for only Win64, leaving everyone else at 2000? Interesting that Linux/BSD etc on AMD64 don't run into this. |
|
|
| msg63540 - (view) |
Author: Amaury Forgeot d'Arc (amaury.forgeotdarc) * |
Date: 2008-03-14 22:49 |
| The situation was the same on win32 with the VC2005 debug build, some months ago. I think this is because of the extra stack checks added with "recent" versions of Visual Studio, together with the fact that local variables are not shared when optimizations are disabled. Can you try to raise the stack size on x64 builds? If 2Mb is enough for 32bit, 4Mb should be good in your case. |
|
|
| msg63902 - (view) |
Author: Trent Nelson (trent) * |
Date: 2008-03-18 07:07 |
| Fixed in r61509; bumped the stack size from 200000 to 210000, which stops the stack overflow from being triggered. |
|
|
| msg147900 - (view) |
Author: Sébastien Sablé (sable) |
Date: 2011-11-18 17:02 |
| I had the exact same problem when compiling the Python 2.7 branch with Visual Studio 2010. In debug mode and 64 bits, python will crash on test_loads_recursion in test_marshal and on the code provided in . Could you please reopen this issue? |
|
|
| msg147907 - (view) |
Author: Sébastien Sablé (sable) |
Date: 2011-11-18 17:18 |
| I don't have the issue anymore when bumping the stack size to 2300000. |
|
|
| msg148367 - (view) |
Author: Terry J. Reedy (terry.reedy) * |
Date: 2011-11-25 23:18 |
| The 'official' PSF build for 2.7 (and 3.x) is done with VS2008, so this may not be the last problem you have. I am pretty sure this will never change for 2.7, but that is not to say we cannot make a simple change to accommodate VS2010. I am sure we will eventually update VS for 3.x. Can you test 3.2 or even better 3.3? |
|
|
| msg148470 - (view) |
Author: Sébastien Sablé (sable) |
Date: 2011-11-28 08:42 |
| Thanks Terry, I am aware of that. We are working on making Python work with VS2010 in issue 13210. I will check with the py3k branch soon and report here if the same problem applies. |
|
|
| msg170651 - (view) |
Author: Trent Nelson (trent) * |
Date: 2012-09-18 15:37 |
| Closing issue; this has been fixed. |
|
|
| msg192211 - (view) |
Author: Richard Oudkerk (sbt) * |
Date: 2013-07-02 17:13 |
| Reopening because I think this is again a problem for Win64 and 3.x. The Win64 buildbots always seem to crash on test_marshal (and I do too). It appears to be BugsTestCase.test_loads_2x_code() which crashes, which is virtually the same as test_loads_recursion(). |
|
|
| msg192213 - (view) |
Author: Richard Oudkerk (sbt) * |
Date: 2013-07-02 17:45 |
| Closing because this is caused by #17206 and is already discussed there. |
|
|