| msg261793 - (view) |
Author: STINNER Victor (vstinner) *  |
Date: 2016-03-15 00:39 |
| Python implements debug hooks on Python memory allocators: https://docs.python.org/dev/c-api/memory.html#c.PyMem_SetupDebugHooks Problem: buffer understand and buffer overflow are only detected when a memory block is released, which may occur far from the code responsible to create the buffer. Attached patch dumps the traceback where a memory block was allocated on error in malloc debug hooks. The feature requires to enable tracemalloc to record tracebacks. test.py used in below example: ---- import _testcapi def f(): _testcapi.pymem_buffer_overflow() f() ---- Example: --- haypo@selma$ ./python -X tracemalloc=5 test.py Debug memory block at address p=0x22e8be0: API 'm' 16 bytes originally requested The 7 pad bytes at p-7 are FORBIDDENBYTE, as expected. The 8 pad bytes at tail=0x22e8bf0 are not all FORBIDDENBYTE (0xfb): at tail+0: 0x78 *** OUCH ... The block was made by call #37240 to debug malloc/realloc. Data at p: cb cb cb cb cb cb cb cb cb cb cb cb cb cb cb cb Memory block traceback (most recent call first): File "test.py", line 4 File "test.py", line 6 Fatal Python error: bad trailing pad byte Current thread 0x00007f4a93d9c700 (most recent call first): File "test.py", line 4 in f File "test.py", line 6 in Abandon (core dumped) --- The patch adds the "Memory block traceback" section. |
|
|
| msg261813 - (view) |
Author: STINNER Victor (vstinner) * |
Date: 2016-03-15 13:17 |
| See also the issue #26567: "Use tracemalloc to display the traceback where an object was allocated when a ResourceWarning is emitted". |
|
|
| msg261816 - (view) |
Author: STINNER Victor (vstinner) * |
Date: 2016-03-15 14:34 |
| I reviewed my own patch on Rietveld :-) |
|
|
| msg261827 - (view) |
Author: Roundup Robot (python-dev)  |
Date: 2016-03-15 20:58 |
| New changeset 18a19e62bac5 by Victor Stinner in branch 'default': Enhance and rewrite traceback dump C functions https://hg.python.org/cpython/rev/18a19e62bac5 New changeset fea3c6e9a38e by Victor Stinner in branch '3.5': _tracemalloc: store lineno as unsigned int https://hg.python.org/cpython/rev/fea3c6e9a38e |
|
|
| msg261828 - (view) |
Author: Roundup Robot (python-dev) |
Date: 2016-03-15 21:42 |
| New changeset cef6a32d805f by Victor Stinner in branch 'default': On memory error, dump the memory block traceback https://hg.python.org/cpython/rev/cef6a32d805f |
|
|
| msg261829 - (view) |
Author: Roundup Robot (python-dev) |
Date: 2016-03-15 21:50 |
| New changeset 8215dae7ec3c by Victor Stinner in branch 'default': Oops, revert unwanted change used to create an example https://hg.python.org/cpython/rev/8215dae7ec3c |
|
|
| msg261830 - (view) |
Author: Roundup Robot (python-dev) |
Date: 2016-03-15 22:47 |
| New changeset 769dfcb701ee by Victor Stinner in branch 'default': Issue #26564: Fix test_capi https://hg.python.org/cpython/rev/769dfcb701ee |
|
|
| msg261841 - (view) |
Author: Martin Panter (martin.panter) * |
Date: 2016-03-16 07:18 |
| FYI Victor if you haven’t already noticed, the Windows buildbots are choking on a variable-length array: ..\Python\traceback.c(513): error C2057: expected constant expression [D:\buildarea\3.x.bolen-windows10\build\PCbuild\pythoncore.vcxproj] ..\Python\traceback.c(513): error C2466: cannot allocate an array of constant size 0 [D:\buildarea\3.x.bolen-windows10\build\PCbuild\pythoncore.vcxproj] ..\Python\traceback.c(513): error C2133: 'buffer': unknown size [D:\buildarea\3.x.bolen-windows10\build\PCbuild\pythoncore.vcxproj] |
|
|
| msg261845 - (view) |
Author: Roundup Robot (python-dev) |
Date: 2016-03-16 08:46 |
| New changeset 5f2284ecf9c6 by Victor Stinner in branch 'default': Fix compilation error of traceback.c on Windows https://hg.python.org/cpython/rev/5f2284ecf9c6 |
|
|
| msg261846 - (view) |
Author: STINNER Victor (vstinner) * |
Date: 2016-03-16 08:51 |
| Martin Panter added the comment: > FYI Victor if you haven’t already noticed, the Windows buildbots are choking on a variable-length array: Thank you, sometimes I miss buildbot failures. I pushed changes which had 2 tests, so I knew that all buildbots were red, and I planned to come back the day after to check. For Windows, it's a pity, the expression is constant. Stupid compiler :-) VS2015 doesn't support C99 yet? Anyway, it's trivial to fix the code. I just fixed it. I will check again if buildbots are now happy with the current code ;-) |
|
|
| msg261849 - (view) |
Author: STINNER Victor (vstinner) * |
Date: 2016-03-16 10:55 |
| Ok, buildbots are green again. I close the issue. |
|
|
| msg261956 - (view) |
Author: Roundup Robot (python-dev) |
Date: 2016-03-18 10:04 |
| New changeset 7b079adb0774 by Victor Stinner in branch 'default': Enhance documentation on malloc debug hooks https://hg.python.org/cpython/rev/7b079adb0774 |
|
|
| msg383236 - (view) |
Author: Irit Katriel (iritkatriel) * |
Date: 2020-12-17 11:50 |
| I'm reopening this to commit a fix for a comment which became obsolete following this change: https://hg.python.org/cpython/rev/18a19e62bac5. |
|
|
| msg383238 - (view) |
Author: STINNER Victor (vstinner) * |
Date: 2020-12-17 12:33 |
| New changeset 40125ab3252453bf205ed906e46bf9741c27bf9d by Irit Katriel in branch 'master': bpo-26564: fix obsolete comment in traceback.c (GH-23819) https://github.com/python/cpython/commit/40125ab3252453bf205ed906e46bf9741c27bf9d |
|
|
| msg383241 - (view) |
Author: miss-islington (miss-islington) |
Date: 2020-12-17 13:19 |
| New changeset cecbaa3a80d5ae28cdd4129d6d2c4395c12a89e4 by Miss Islington (bot) in branch '3.8': bpo-26564: fix obsolete comment in traceback.c (GH-23819) https://github.com/python/cpython/commit/cecbaa3a80d5ae28cdd4129d6d2c4395c12a89e4 |
|
|
| msg383242 - (view) |
Author: miss-islington (miss-islington) |
Date: 2020-12-17 13:20 |
| New changeset 1c70d40e94741578ce28b8851fb65372ac2e7942 by Miss Islington (bot) in branch '3.9': bpo-26564: fix obsolete comment in traceback.c (GH-23819) https://github.com/python/cpython/commit/1c70d40e94741578ce28b8851fb65372ac2e7942 |
|
|
| msg383244 - (view) |
Author: Serhiy Storchaka (serhiy.storchaka) * |
Date: 2020-12-17 13:48 |
| I am wondering why 53/22? 5/2 is as good for 8-bytes long -- both gives 21-bytes buffer. I do not propose to change anything. It is just curiosity. |
|
|
| msg383246 - (view) |
Author: STINNER Victor (vstinner) * |
Date: 2020-12-17 13:58 |
| > I am wondering why 53/22? 5/2 is as good for 8-bytes long -- both gives 21-bytes buffer. 53/22 is closer to math.log(256)/math.log(10) than 5/2, and 8*53 should not overflow :-) |
|
|