bpo-34395: Don't free allocated memory on realloc fail in load_mark() in _pickle.c. by sir-sigurd · Pull Request #8788 · python/cpython (original) (raw)

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service andprivacy statement. We’ll occasionally send you account related emails.

Already on GitHub?Sign in to your account

Conversation15 Commits3 Checks0 Files changed

Conversation

This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.Learn more about bidirectional Unicode characters

[ Show hidden characters]({{ revealButtonHref }})

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Seems the check above can be removed. alloc > (PY_SSIZE_T_MAX / sizeof(Py_ssize_t)) is duplicated in PyMem_RESIZE(), and failing alloc <= ((size_t)self->num_marks + 1) is not possible if sizeof(Py_ssize_t) > 2.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also, it seems that new space is needed only when self->num_marks + 1 is strictly greater than self->marks_size.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agree. This condition can be written as self->num_marks >= self->marks_size or self->num_marks == self->marks_size.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@serhiy-storchaka
I'd prefer to make these change as separate commits/RPs as they are not related to this bpo issue.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PR for upsizing issue #8860.

Need backport to make code consistent or as enhancement?

I don't think there is a bug that needs to be fixed. This change looks rather as a code clean up to me. If I'm wrong, and it fixes a real bug, it should be backported.

@sir-sigurd, please remove also the redundant check before PyMem_RESIZE if it is truly unneeded.

@serhiy-storchaka

I'm considering adding this as explanation why overflow check is not needed

assert((size_t)self->num_marks <= PY_SSIZE_T_MAX / sizeof(Py_ssize_t));
Py_BUILD_ASSERT((size_t)PY_SSIZE_T_MAX / sizeof(Py_ssize_t) << 1 <= SIZE_MAX - 20);

but I'm not sure if it makes things clear.

I think this is not needed. Just remove the check.

failing alloc <= ((size_t)self->num_marks + 1) is not possible if sizeof(Py_ssize_t) > 2.

@serhiy-storchaka Can this be guaranteed in CPython? Wikipedia says size_t is at least 16 bit wise.

CuriousLearner added a commit to CuriousLearner/cpython that referenced this pull request

Aug 27, 2018

• master: (104 commits) Fast path for exact floats in math.hypot() and math.dist() (pythonGH-8949) Remove AIX workaround test_subprocess (pythonGH-8939) bpo-34503: Fix refleak in PyErr_SetObject() (pythonGH-8934) closes bpo-34504: Remove the useless NULL check in PySequence_Check(). (pythonGH-8935) closes bpo-34501: PyType_FromSpecWithBases: Check spec->name before dereferencing it. (pythonGH-8930) closes bpo-34502: Remove a note about utf8_mode from sys.exit() docs. (pythonGH-8928) Remove unneeded PyErr_Clear() in _winapi_SetNamedPipeHandleState_impl() (pythonGH-8281) Fix markup in stdtypes documentation (pythonGH-8905) bpo-34395: Don't free allocated memory on realloc fail in load_mark() in _pickle.c. (pythonGH-8788) Fix upsizing of marks stack in pickle module. (pythonGH-8860) bpo-34171: Prevent creating Lib/trace.cover when run the trace module. (pythonGH-8841) closes bpo-34493: Objects/genobject.c: Add missing NULL check to compute_cr_origin() (pythonGH-8911) Fixed typo with asynccontextmanager code example (pythonGH-8845) bpo-34426: fix typo (lltrace -> ltrace) (pythonGH-8822) bpo-13312: Avoid int underflow in time year. (pythonGH-8912) bpo-34492: Python/coreconfig.c: Fix _Py_wstrlist_copy() (pythonGH-8910) bpo-34448: Improve output of usable wchar_t check (pythonGH-8846) closes bpo-34471: _datetime: Add missing NULL check to tzinfo_from_isoformat_results. (pythonGH-8869) bpo-6700: Fix inspect.getsourcelines for module level frames/tracebacks (pythonGH-8864) Fix typo in the dataclasses's doc (pythonGH-8896) ...

I think we can assume that size_t is at least 32 bit wise, and char is 8 bit wise. Many things in CPython implementation will be broken if this is not true.

If I'm not mistaken, it can't overflow even if sizeof(Py_ssize_t) == 2 (presuming that CPython allocates at most PY_SSIZE_T_MAX bytes).

If I'm not mistaken, it can't overflow even if sizeof(Py_ssize_t) == 2 (presuming that CPython allocates at most PY_SSIZE_T_MAX bytes).

I don't get it, how it's not possible.

And rechecking the patch it seems to me the overflow check is not right. You should check overflow before doing math operation. size_t alloc = ((size_t)self->num_marks << 1) + 20; might already overflowed and succeed in the overflow check in PyMem_Resize. What do you think about it? Am I mistaken something?

@zhangyangyu
Presuming that CPython allocates at most PY_SSIZE_T_MAX bytes, we can assume that (size_t)self->num_marks <= PY_SSIZE_T_MAX / sizeof(Py_ssize_t). So
alloc <= (PY_SSIZE_T_MAX / sizeof(Py_ssize_t) << 1) + 20. If sizeof(Py_ssize_t) == 2, then alloc <= PY_SSIZE_T_MAX + 20 and it's obviously less than SIZE_MAX.

Makes sense?