Issue 34087: int(s), float(s) and others may cause segmentation fault (original) (raw)

Created on 2018-07-10 19:36 by fenrrir, last changed 2022-04-11 14:59 by admin. This issue is now closed.

Messages (34)

msg321394 - (view)

Author: Rodrigo Pinheiro Marques de Araújo (fenrrir)

Date: 2018-07-10 19:36

msg321404 - (view)

Author: Inada Naoki (methane) * (Python committer)

Date: 2018-07-11 01:02

It's difficult to investigate without reproducible example.

Do you use 3rd party extension module? Typical cause of these crash is bug in 3rd party extensions.

msg321406 - (view)

Author: Rodrigo Pinheiro Marques de Araújo (fenrrir)

Date: 2018-07-11 01:39

Sorry for that. I’m not able to make a little example to reproduce this bug. It’s happens during Django tests on a very large code base. A interest thing is that not happens with “-X dev” parameters. Please, any suggestions how I can get more details about this bug?

msg321407 - (view)

Author: Inada Naoki (methane) * (Python committer)

Date: 2018-07-11 03:00

If example cannot be small, you can share the project on Github or other place. If your project is private, you can create OSS version of reproducer. If you can't create OSS reproducer, we can't debug it.

3rd party library list may helpful, but I'm not sure.

msg321450 - (view)

Author: Rodrigo Pinheiro Marques de Araújo (fenrrir)

Date: 2018-07-11 13:27

Running with -X faulthandler

Fatal Python error: Segmentation fault

Current thread 0x00007fff89cf2380 (most recent call first): File "/Users/rodrigo/root/lib/python3.7/site-packages/django/urls/resolvers.py", line 526 in resolve File "/Users/rodrigo/root/lib/python3.7/site-packages/django/urls/resolvers.py", line 500 in resolve File "/Users/rodrigo/root/lib/python3.7/site-packages/django/core/handlers/base.py", line 113 in _get_response File "/Users/rodrigo/root/lib/python3.7/site-packages/django/core/handlers/exception.py", line 35 in inner File "/Users/rodrigo/root/lib/python3.7/site-packages/reversion/views.py", line 43 in do_revision_view File "/Users/rodrigo/root/lib/python3.7/site-packages/reversion/middleware.py", line 51 in call File "/Users/rodrigo/root/lib/python3.7/site-packages/django/core/handlers/exception.py", line 35 in inner File "/Users/rodrigo/root/lib/python3.7/site-packages/django/utils/deprecation.py", line 95 in call File "/Users/rodrigo/root/lib/python3.7/site-packages/django/core/handlers/exception.py", line 35 in inner File "/Users/rodrigo/root/lib/python3.7/site-packages/django/utils/deprecation.py", line 95 in call File "/Users/rodrigo/root/lib/python3.7/site-packages/django/core/handlers/exception.py", line 35 in inner File "/Users/rodrigo/root/lib/python3.7/site-packages/django/utils/deprecation.py", line 95 in call File "/Users/rodrigo/root/lib/python3.7/site-packages/django/core/handlers/exception.py", line 35 in inner File "/Users/rodrigo/root/lib/python3.7/site-packages/django/utils/deprecation.py", line 95 in call File "/Users/rodrigo/root/lib/python3.7/site-packages/django/core/handlers/exception.py", line 35 in inner File "/Users/rodrigo/root/lib/python3.7/site-packages/django/utils/deprecation.py", line 95 in call File "/Users/rodrigo/root/lib/python3.7/site-packages/django/core/handlers/exception.py", line 35 in inner File "/Users/rodrigo/root/lib/python3.7/site-packages/django/utils/deprecation.py", line 95 in call File "/Users/rodrigo/root/lib/python3.7/site-packages/django/core/handlers/exception.py", line 35 in inner File "/Users/rodrigo/root/lib/python3.7/site-packages/django/utils/deprecation.py", line 95 in call File "/Users/rodrigo/root/lib/python3.7/site-packages/django/core/handlers/exception.py", line 35 in inner File "/Users/rodrigo/root/lib/python3.7/site-packages/django/utils/deprecation.py", line 95 in call File "/Users/rodrigo/root/lib/python3.7/site-packages/django/core/handlers/exception.py", line 35 in inner File "/Users/rodrigo/root/lib/python3.7/site-packages/django/utils/deprecation.py", line 95 in call File "/Users/rodrigo/root/lib/python3.7/site-packages/django/core/handlers/exception.py", line 35 in inner File "/Users/rodrigo/root/lib/python3.7/site-packages/django/utils/deprecation.py", line 95 in call File "/Users/rodrigo/root/lib/python3.7/site-packages/django/core/handlers/exception.py", line 35 in inner File "/Users/rodrigo/root/lib/python3.7/site-packages/django/utils/deprecation.py", line 95 in call File "/Users/rodrigo/root/lib/python3.7/site-packages/django/core/handlers/exception.py", line 35 in inner File "/Users/rodrigo/root/lib/python3.7/site-packages/django/utils/deprecation.py", line 95 in call File "/Users/rodrigo/root/lib/python3.7/site-packages/django/core/handlers/exception.py", line 35 in inner File "/Users/rodrigo/root/lib/python3.7/site-packages/django/core/handlers/base.py", line 81 in get_response File "/Users/rodrigo/root/lib/python3.7/site-packages/django/test/client.py", line 138 in call File "/Users/rodrigo/root/lib/python3.7/site-packages/django/test/client.py", line 467 in request File "/Users/rodrigo/root/lib/python3.7/site-packages/django/test/client.py", line 404 in generic File "/Users/rodrigo/root/lib/python3.7/site-packages/django/test/client.py", line 332 in get File "/Users/rodrigo/root/lib/python3.7/site-packages/django/test/client.py", line 517 in get File "/Users/rodrigo/root/lib/python3.7/unittest/case.py", line 615 in run File "/Users/rodrigo/root/lib/python3.7/unittest/case.py", line 663 in call File "/Users/rodrigo/root/lib/python3.7/site-packages/django/test/testcases.py", line 206 in call File "/Users/rodrigo/root/lib/python3.7/unittest/suite.py", line 122 in run File "/Users/rodrigo/root/lib/python3.7/unittest/suite.py", line 84 in call File "/Users/rodrigo/root/lib/python3.7/unittest/runner.py", line 176 in run File "/Users/rodrigo/root/lib/python3.7/site-packages/django/test/runner.py", line 569 in run_suite File "/Users/rodrigo/root/lib/python3.7/site-packages/django/test/runner.py", line 603 in run_tests File "/Users/rodrigo/root/lib/python3.7/site-packages/django/core/management/commands/test.py", line 59 in handle File "/Users/rodrigo/root/lib/python3.7/site-packages/django/core/management/base.py", line 335 in execute File "/Users/rodrigo/root/lib/python3.7/site-packages/django/core/management/base.py", line 288 in run_from_argv File "/Users/rodrigo/root/lib/python3.7/site-packages/django/core/management/commands/test.py", line 26 in run_from_argv File "/Users/rodrigo/root/lib/python3.7/site-packages/django/core/management/init.py", line 365 in execute File "/Users/rodrigo/root/lib/python3.7/site-packages/django/core/management/init.py", line 371 in execute_from_command_line File "manage.py", line 10 in Segmentation fault: 11

The Django code:

def resolve(self, path):
    path = str(path)  # path may be a reverse_lazy object
    tried = []
    match = self.pattern.match(path)
    if match:
        new_path, args, kwargs = match
        for pattern in self.url_patterns:
            try:
                sub_match = pattern.resolve(new_path)
            except Resolver404 as e:
                sub_tried = e.args[0].get('tried')
                if sub_tried is not None:
                    tried.extend([pattern] + t for t in sub_tried)
                else:
                    tried.append([pattern])
            else:
                if sub_match:
                    # Merge captured arguments in match with submatch
                    sub_match_dict = dict(kwargs, **self.default_kwargs)
                    # Update the sub_match_dict with the kwargs from the sub_match.
                    sub_match_dict.update(sub_match.kwargs)
                    # If there are *any* named groups, ignore all non-named groups.
                    # Otherwise, pass all non-named arguments as positional arguments.
                    sub_match_args = sub_match.args
                    if not sub_match_dict:
                        sub_match_args = args + sub_match.args
                    return ResolverMatch(
                        sub_match.func,
                        sub_match_args,
                        sub_match_dict,
                        sub_match.url_name,
                        [self.app_name] + sub_match.app_names,
                        [self.namespace] + sub_match.namespaces,
                    )
                tried.append([pattern]) # Segfault happens here!
        raise Resolver404({'tried': tried, 'path': new_path})
    raise Resolver404({'path': path})

According to faulthandler the segmentation fault happens on tried.append([pattern]).

msg321452 - (view)

Author: STINNER Victor (vstinner) * (Python committer)

Date: 2018-07-11 14:03

Please try to set PYTHONMALLOC=debug environment variable and run again your test. I hope that Python will be able to catch a memory corruption somewhere. If it's the case, you can try:

PYTHONMALLOC=debug python3 -X tracemalloc=10 (...)

To see where the corrupted object has been allocated.

msg321491 - (view)

Author: Rodrigo Pinheiro Marques de Araújo (fenrrir)

Date: 2018-07-11 18:02

Unfortunately with 'PYTHONMALLOC=debug' the segmentation fault do not happen.

msg321533 - (view)

Author: Inada Naoki (methane) * (Python committer)

Date: 2018-07-12 10:04

Would you provide list of extension modules? What happens when removing some of them?

msg321560 - (view)

Author: Jonas H. (jonash) *

Date: 2018-07-12 14:54

I also have a segfault that goes away with malloc debugging. Not sure if it's the same issue.

My extension modules are

venv/lib/python3.7/site-packages//_yaml.cpython-37m-darwin.so venv/lib/python3.7/site-packages//netifaces.cpython-37m-darwin.so venv/lib/python3.7/site-packages//PIL/_webp.cpython-37m-darwin.so venv/lib/python3.7/site-packages//PIL/_imagingft.cpython-37m-darwin.so venv/lib/python3.7/site-packages//PIL/_imagingcms.cpython-37m-darwin.so venv/lib/python3.7/site-packages//PIL/_imaging.cpython-37m-darwin.so venv/lib/python3.7/site-packages//PIL/_imagingmath.cpython-37m-darwin.so venv/lib/python3.7/site-packages//PIL/_imagingtk.cpython-37m-darwin.so venv/lib/python3.7/site-packages//PIL/_imagingmorph.cpython-37m-darwin.so venv/lib/python3.7/site-packages//lxml/builder.cpython-37m-darwin.so venv/lib/python3.7/site-packages//lxml/_elementpath.cpython-37m-darwin.so venv/lib/python3.7/site-packages//lxml/html/diff.cpython-37m-darwin.so venv/lib/python3.7/site-packages//lxml/html/clean.cpython-37m-darwin.so venv/lib/python3.7/site-packages//lxml/etree.cpython-37m-darwin.so venv/lib/python3.7/site-packages//lxml/objectify.cpython-37m-darwin.so venv/lib/python3.7/site-packages//coverage/tracer.cpython-37m-darwin.so

Unfortunately I can't test the application without any of these, but maybe we can match my list with fenrrir's.

msg321561 - (view)

Author: Jonas H. (jonash) *

Date: 2018-07-12 14:54

Btw my segfault is from Django too, but that may just be a coincidence

msg321566 - (view)

Author: Rodrigo Pinheiro Marques de Araújo (fenrrir)

Date: 2018-07-12 15:31

Extensions:

./_cffi_backend.cpython-37m-darwin.so ./_mssql.cpython-37m-darwin.so ./_yaml.cpython-37m-darwin.so ./Crypto/Cipher/_AES.cpython-37m-darwin.so ./Crypto/Cipher/_ARC2.cpython-37m-darwin.so ./Crypto/Cipher/_ARC4.cpython-37m-darwin.so ./Crypto/Cipher/_Blowfish.cpython-37m-darwin.so ./Crypto/Cipher/_CAST.cpython-37m-darwin.so ./Crypto/Cipher/_DES.cpython-37m-darwin.so ./Crypto/Cipher/_DES3.cpython-37m-darwin.so ./Crypto/Cipher/_XOR.cpython-37m-darwin.so ./Crypto/Hash/_MD2.cpython-37m-darwin.so ./Crypto/Hash/_MD4.cpython-37m-darwin.so ./Crypto/Hash/_RIPEMD160.cpython-37m-darwin.so ./Crypto/Hash/_SHA224.cpython-37m-darwin.so ./Crypto/Hash/_SHA256.cpython-37m-darwin.so ./Crypto/Hash/_SHA384.cpython-37m-darwin.so ./Crypto/Hash/_SHA512.cpython-37m-darwin.so ./Crypto/PublicKey/_fastmath.cpython-37m-darwin.so ./Crypto/Util/_counter.cpython-37m-darwin.so ./Crypto/Util/strxor.cpython-37m-darwin.so ./cryptography/hazmat/bindings/_constant_time.abi3.so ./cryptography/hazmat/bindings/_openssl.abi3.so ./cryptography/hazmat/bindings/_padding.abi3.so ./Cython/Compiler/Code.cpython-37m-darwin.so ./Cython/Compiler/FlowControl.cpython-37m-darwin.so ./Cython/Compiler/FusedNode.cpython-37m-darwin.so ./Cython/Compiler/Lexicon.cpython-37m-darwin.so ./Cython/Compiler/Parsing.cpython-37m-darwin.so ./Cython/Compiler/Pythran.cpython-37m-darwin.so ./Cython/Compiler/Scanning.cpython-37m-darwin.so ./Cython/Compiler/Visitor.cpython-37m-darwin.so ./Cython/Plex/Actions.cpython-37m-darwin.so ./Cython/Plex/Scanners.cpython-37m-darwin.so ./Cython/Runtime/refnanny.cpython-37m-darwin.so ./Cython/StringIOTree.cpython-37m-darwin.so ./Cython/Tempita/_tempita.cpython-37m-darwin.so ./lazy_object_proxy/cext.cpython-37m-darwin.so ./lxml/_elementpath.cpython-37m-darwin.so ./lxml/builder.cpython-37m-darwin.so ./lxml/etree.cpython-37m-darwin.so ./lxml/html/clean.cpython-37m-darwin.so ./lxml/html/diff.cpython-37m-darwin.so ./lxml/objectify.cpython-37m-darwin.so ./markupsafe/_speedups.cpython-37m-darwin.so ./PIL/_imaging.cpython-37m-darwin.so ./PIL/_imagingcms.cpython-37m-darwin.so ./PIL/_imagingmath.cpython-37m-darwin.so ./PIL/_imagingmorph.cpython-37m-darwin.so ./PIL/_imagingtk.cpython-37m-darwin.so ./psycopg2/_psycopg.cpython-37m-darwin.so ./pymssql.cpython-37m-darwin.so ./reportlab/graphics/_renderPM.cpython-37m-darwin.so ./reportlab/lib/_rl_accel.cpython-37m-darwin.so ./setproctitle.cpython-37m-darwin.so ./simplejson/_speedups.cpython-37m-darwin.so ./wrapt/_wrappers.cpython-37m-darwin.so

requirements.txt

alabaster==0.7.11 amqp==2.2.2 appnope==0.1.0 arrow==0.12.0 asn1crypto==0.24.0 astroid==1.6.5 Babel==2.6.0 beautifulsoup4==4.6.0 billiard==3.5.0.3 boto3==1.7.24 botocore==1.10.24 celery==4.1.0 certifi==2018.4.16 cffi==1.11.2 chardet==3.0.4 colorama==0.3.9 colorful==0.4.0 configparser==3.5.0 coreapi==2.3.3 coreschema==0.0.4 cryptography==2.2.2 cssselect==1.0.3 Cython==0.28.2 decorator==4.3.0 dj-database-url==0.4.2 Django==2.0.6 django-braces==1.12.0 django-celery-beat==1.1.1 django-celery-results==1.0.1 django-ckeditor==5.2.1 django-colorful==1.2 django-crispy-forms==1.6.1 django-extensions==1.9.8 django-extra-views==0.9.0 django-filter==1.0.4 django-filters==0.2.1 django-formset-js==0.5.0 django-formtools==2.1 django-fsm==2.6.0 django-jquery-js==3.1.1 django-js-asset==1.1.0 django-localflavor==1.6.2 django-model-utils==3.1.1 django-mptt==0.9.0 django-pagination==1.0.10 django-querysetsequence==0.8 django-recaptcha2==1.0.3 django-rest-swagger==2.1.2 django-reversion==2.0.13 django-reversion-compare==0.8.4 django-storages==1.6.6 django-tables2==1.17.1 django-tables2-reports==0.1.3 django-taggit==0.22.1 django-wkhtmltopdf==3.1.0 djangorestframework==3.7.3 djangorestframework-filters==0.10.2 djangorestframework-jwt==1.11.0 docutils==0.14 feedparser==5.2.1 google-api-python-client==1.6.4 gunicorn==19.7.1 html5lib==1.0.1 httplib2==0.10.3 idna==2.6 imagesize==0.7.1 ipaddress==1.0.19 ipdb==0.10.3 ipython==6.2.1 ipython-genutils==0.2.0 isort==4.2.15 itypes==1.1.0 jedi==0.12.1 Jinja2==2.10 jmespath==0.9.3 kombu==4.2.1 lazy-object-proxy==1.3.1 ldap3==2.4 lxml==4.1.1 MarkupSafe==1.0 mccabe==0.6.1 MechanicalSoup==0.10.0 minio==4.0.0 model-mommy==1.5.0 oauth2client==4.1.2 olefile==0.45.1 openapi-codec==1.3.2 parso==0.3.1 pathlib2==2.3.0 pbr==3.1.1 pdfrw==0.4 pexpect==4.3.1 pickleshare==0.7.4 Pillow==4.3.0 prompt-toolkit==1.0.15 psycopg2==2.7.3.2 ptyprocess==0.5.2 pyasn1==0.4.2 pyasn1-modules==0.2.1 pycparser==2.18 pycpfcnpj==1.2 pycrypto==2.6.1 Pygments==2.2.0 PyJWT==1.6.4 pylint==1.8.1 pyOpenSSL==17.5.0 PyPDF2==1.26.0 pyquery==1.3.0 python-dateutil==2.6.1 python-logstash==0.4.6 python-magic==0.4.15 python-memcached==1.58 pytz==2018.4 PyYAML==3.13 qrcode==5.3 raven==6.4.0 reportlab==3.4.0 requests==2.18.4 rsa==3.4.2 s3transfer==0.1.13 setproctitle==1.1.10 simplegeneric==0.8.1 simplejson==3.13.2 six==1.11.0 snowballstemmer==1.2.1 Sphinx==1.6.5 sphinxcontrib-websupport==1.1.0 suds-py3==1.3.3.0 traitlets==4.3.2 typing==3.6.4 Unidecode==0.4.21 uritemplate==3.0.0 urllib3==1.22 vine==1.1.4 wcwidth==0.1.7 webencodings==0.5.1 wrapt==1.10.11 xlrd==1.1.0 xlutils==2.0.0 xlwt==1.3.0 xmljson==0.1.9

msg321567 - (view)

Author: Inada Naoki (methane) * (Python committer)

Date: 2018-07-12 15:32

Hmm, all segfault are happend on macOS?

msg321577 - (view)

Author: Jonas H. (jonash) *

Date: 2018-07-12 19:07

I can reproduce this on Ubuntu 18.04.

INADA, I have a full gdb backtrace with Python 3.7 development build. I'd like to share it with you privately as I'm concerned it may contain sensible information. I know that's a bit unconventional; if you have other suggestions I'm happy to follow along.

msg321578 - (view)

Author: Rodrigo Pinheiro Marques de Araújo (fenrrir)

Date: 2018-07-12 19:14

I did remove PyYAML, lxml and Pillow here but segfault still happen

msg321581 - (view)

Author: Inada Naoki (methane) * (Python committer)

Date: 2018-07-12 22:18

@jonash, Thank you. Would you share core file too? You can get core file when reproducing it after ulimit -c unlimited.

My mail address is "songofacandy [at] gmail.com". Please share it via Dropbox or Google Drive.

msg321586 - (view)

Author: Inada Naoki (methane) * (Python committer)

Date: 2018-07-13 09:42

I received two core files ("core" and "core.23") from fenrrir, thank you. But I can't investigate well, sorry. I think someone breaks memory, and SEGV happened later. I need to reproduce it on my machine...

In core.23, SEGV happens in kill() syscall. I can't get how it can happened.

(gdb) bt 10 #0 0x00007ffff6c10187 in kill () at ../sysdeps/unix/syscall-template.S:78 #1 0x0000555555778f71 in os_kill_impl (module=, signal=, pid=) at ../Modules/posixmodule.c:6749 #2 os_kill (module=, args=, nargs=) at ../Modules/clinic/posixmodule.c.h:2672 #3 0x00005555555dd4ba in _PyMethodDef_RawFastCallKeywords (kwnames=0x0, nargs=140737353236984, args=0x7ffff7ee7e60, self=<module at remote 0x7ffff7f221d8>, method=0x555555c007a0 <posix_methods+1920>) at ../Objects/call.c:651

In "core", SEGV happens in pymalloc. Pool is broken and I suppose some buffer overflow breaks the pool, but I'm not sure.

#0 pymalloc_alloc (ptr_p=ptr_p@entry=0x7fffee81a640, nbytes=40, ctx=) at ../Objects/obmalloc.c:1398 #1 0x000055555562ed78 in pymalloc_alloc (ctx=, nbytes=40, ptr_p=0x7fffee81a640) at ../Objects/obmalloc.c:1565 #2 _PyObject_Malloc (ctx=, nbytes=40) at ../Objects/obmalloc.c:1555 #3 0x000055555561d5d5 in new_dict_with_shared_keys (keys=0x7ffff4e4c2d0) at ../Objects/dictobject.c:607 #4 _PyObjectDict_SetItem (tp=tp@entry=0x555555e8b338, dictptr=0x7fffedfe4250, key=key@entry='s', value=value@entry='\n

') at ../Objects/dictobject.c:4311 #5 0x000055555562b2d4 in _PyObject_GenericSetAttrWithDict (dict=0x0, value='\n

', name='s', obj=<TextNode at remote 0x7fffedfe4240>) at ../Objects/object.c:1319 #6 PyObject_GenericSetAttr (obj=<TextNode at remote 0x7fffedfe4240>, name='s', value='\n

') at ../Objects/object.c:1341 #7 0x000055555562a09f in PyObject_SetAttr (v=v@entry=<TextNode at remote 0x7fffedfe4240>, name=, value=value@entry='\n

') at ../Objects/object.c:978 #8 0x00005555555c67d8 in _PyEval_EvalFrameDefault (f=, throwflag=) at ../Python/ceval.c:2002 #9 0x00005555555c1787 in function_code_fastcall (co=co@entry=0x7ffff4e90930, args=, args@entry=0x7fffee81a930, nargs=2, globals=globals@entry={'name': 'django.template.base', 'doc': "\nThis is the Django template system... ../Objects/call.c:283

(gdb) p pool $1 = (poolp) 0x7fffee861000 (gdb) p bp $2 = (block *) 0x7a672f <error: Cannot access memory at address 0x7a672f> (gdb) p *pool $3 = {ref = {_padding = 0x2c <error: Cannot access memory at address 0x2c>, count = 44}, freeblock = 0x7a672f <error: Cannot access memory at address 0x7a672f>, nextpool = 0x555555ba1fd0 <usedpools+48>, prevpool = 0x555555ba1fd0 <usedpools+48>, arenaindex = 170, szidx = 4, nextoffset = 1768, maxnextoffset = 4056} (gdb) p *(pool->nextpool) $4 = {ref = {_padding = 0x7fffeec2a000 "}", count = 4005732352}, freeblock = 0x7fffeea8e000 "\006", nextpool = 0x7fffee861000, prevpool = 0x7fffee861000, arenaindex = 4004200448, szidx = 32767, nextoffset = 4004200448, maxnextoffset = 32767}

msg321588 - (view)

Author: Jonas H. (jonash) *

Date: 2018-07-13 10:41

Reduced it to something that seems unicode related?

No extension modules involved. Vanilla Django project with a single url + template.

See testproj/urls.py and tmpl/index.html

msg321589 - (view)

Author: STINNER Victor (vstinner) * (Python committer)

Date: 2018-07-13 10:42

If someone expects progress on these issues, we need to get a repository that we can clone to reproduce the bug. Then the code must be simplified as much as possible to reduce the code base. The best is when you still trigger the crash with less than 1000 lines of code and no or very few dependencies.

msg321590 - (view)

Author: Inada Naoki (methane) * (Python committer)

Date: 2018-07-13 10:57

@jonash Thanks! Would you give me concrete step to run the project and reproduce the segfault?

msg321591 - (view)

Author: Jonas H. (jonash) *

Date: 2018-07-13 11:00

Sure.

Unpack archive, create new 3.7 venv with Django (latest version is fine), ./manage.py runserver, curl localhost:8000.

msg321593 - (view)

Author: Inada Naoki (methane) * (Python committer)

Date: 2018-07-13 11:14

I tried it but I can't reproduce...

msg321594 - (view)

Author: Inada Naoki (methane) * (Python committer)

Date: 2018-07-13 11:30

@jonash What Python binary do you use?

msg321596 - (view)

Author: Jonas H. (jonash) *

Date: 2018-07-13 11:48

Here's a Docker image that reproduces the bug.

FROM ubuntu:18.04 RUN apt update && apt install -y python3.7-dbg python3.7-venv python3-venv wget RUN python3.7 -m venv venv RUN venv/bin/pip install django RUN wget https://bugs.python.org/file47688/testproj.tar.gz -O - | tar xfz - CMD cd /testproj && /venv/bin/python manage.py runserver & sleep 5; wget -t1 localhost:8000 >/dev/null 2>/dev/null

Of course this also works outside Docker. I have reproduced with macOS 10.13.5 (17F77) and Ubuntu 18.04 (Docker).

On macOS it's Python 3.7.0 (default, Jun 29 2018, 20:13:13) [Clang 9.1.0 (clang-902.0.39.2)] on darwin, installed from Homebrew.

On Ubuntu it's whatever is in the 18.04 repositories.

msg321602 - (view)

Author: Rodrigo Pinheiro Marques de Araújo (fenrrir)

Date: 2018-07-13 12:50

I can reproduce the segmentation fault using 'testproj.tar.gz' with homebrew and compiled from source. MacOS X High Sierra 10.13.5 (17F77).

msg321603 - (view)

Author: Inada Naoki (methane) * (Python committer)

Date: 2018-07-13 12:56

Very thanks! I found I set export PYTHONMALLOC=pymalloc_debug. That's why I can't reproduce it.

Now I can reproduce and I get where memory is broken, and understood what cause:

Python traceback is: Traceback (most recent call first): File "/home/inada-n/t/bpo34087/testproj/v38/lib/python3.8/site-packages/django/template/base.py", line 767, in init self.literal = float(var) File "/home/inada-n/t/bpo34087/testproj/v38/lib/python3.8/site-packages/django/template/base.py", line 646, in init var_obj = Variable(constant).resolve({}) File "/home/inada-n/t/bpo34087/testproj/v38/lib/python3.8/site-packages/django/template/base.py", line 568, in compile_filter return FilterExpression(token, self) File "/home/inada-n/t/bpo34087/testproj/v38/lib/python3.8/site-packages/django/templatetags/i18n.py", line 368, in do_translate message_string = parser.compile_filter(bits[1]) File "/home/inada-n/t/bpo34087/testproj/v38/lib/python3.8/site-packages/django/template/base.py", line 481, in parse compiled_result = compile_func(self, token) File "/home/inada-n/t/bpo34087/testproj/v38/lib/python3.8/site-packages/django/template/loader_tags.py", line 233, in do_block nodelist = parser.parse(('endblock',)) File "/home/inada-n/t/bpo34087/testproj/v38/lib/python3.8/site-packages/django/template/base.py", line 481, in parse compiled_result = compile_func(self, token) File "/home/inada-n/t/bpo34087/testproj/v38/lib/python3.8/site-packages/django/template/base.py", line 198, in compile_nodelist return parser.parse() File "/home/inada-n/t/bpo34087/testproj/v38/lib/python3.8/site-packages/django/template/base.py", line 160, in init self.nodelist = self.compile_nodelist() File "/home/inada-n/t/bpo34087/testproj/v38/lib/python3.8/site-packages/django/template/loaders/base.py", line 30, in get_template

C traceback is:

#0 0x0000557880378c95 in _Py_string_to_number_with_underscores (s=0x7ff07d8e7bc0 ""So erstellen Sie einen Zahlungslink f?ded_blocks'", orig_len=39, what=0x55788046d60a "float", obj='"So erstellen Sie einen Zahlungslink für Ihre Rechnung"', arg=0x7ff07d9368b0, innerfunc=0x557880276b10 ) at Python/pystrtod.c:410 #1 0x0000557880279b23 in PyFloat_FromString (v='"So erstellen Sie einen Zahlungslink für Ihre Rechnung"') at Objects/floatobject.c:208 #2 0x0000557880279cd5 in float_new_impl (x=, type=0x5578806e4ee0 ) at Objects/floatobject.c:1627 #3 float_new (type=, args=('"So erstellen Sie einen Zahlungslink für Ihre Rechnung"',), kwargs=) at Objects/clinic/floatobject.c.h:181 #4 0x00005578802cfdc5 in type_call (type=type@entry=0x5578806e4ee0 , args=args@entry=('"So erstellen Sie einen Zahlungslink für Ihre Rechnung"',), kwds=kwds@entry=0x0) at Objects/typeobject.c:929 #5 0x000055788026b3bf in _PyObject_FastCallKeywords (callable=callable@entry=<type at remote 0x5578806e4ee0>, stack=, nargs=nargs@entry=1, kwnames=kwnames@entry=0x0) at Objects/call.c:199 #6 0x00005578802526f4 in call_function (kwnames=0x0, oparg=, pp_stack=) at Python/ceval.c:4600 #7 _PyEval_EvalFrameDefault (f=, throwflag=) at Python/ceval.c:3183 #8 0x000055788024fb17 in function_code_fastcall (co=co@entry=0x7ff07ec4f0c0, args=, args@entry=0x7ffc786f2330, nargs=2, globals=globals@entry={'name': 'django.template.base', 'doc': "\nThis is the Django template system.\n\nHow it works:\n\nThe Lexer.tokenize() function converts a template string (i.e., a string containing\nmarkup with custom template tags) to tokens, which can be either plain text\n(TOKEN_TEXT), variables (TOKEN_VAR) or block statements (TOKEN_BLOCK).\n\nThe Parser( ) class takes a list of tokens in its constructor, and its parse()\nmethod returns a compiled template -- which is, under the hood, a list of\nNode objects.\n\nEach Node is responsible for c reating some sort of output -- e.g. simple text\n(TextNode), variable values in a given context (VariableNode), results of basic\nlogic (IfNode), results of looping (ForNode), or anything el se. The core Node\ntypes are TextNode, VariableNode, IfNode and ForNode, but plugin modules can\ndefine their own custom node types.\n\nEach Node has a render() method, which takes a Context and returns a string of\nthe rendered node. For example, the render() method of a Variable Node ret...(truncated)) at Objects/call.c:283 #9 0x000055788026ad87 in _PyFunction_FastCallDict (func=<function at remote 0x7ff07ec2f1e0>, args=0x7ffc786f2330, nargs=, kwargs=0x0) at Objects/call.c:322 #10 0x000055788026ddc1 in _PyObject_FastCallDict (kwargs=0x0, nargs=2, args=0x7ffc786f2330, callable=<function at remote 0x7ff07ec2f1e0>) at Objects/call.c:98

msg321605 - (view)

Author: STINNER Victor (vstinner) * (Python committer)

Date: 2018-07-13 13:00

It might be a duplicate of bpo-33954: bug in _PyUnicode_InsertThousandsGrouping().

msg321613 - (view)

Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer)

Date: 2018-07-13 13:57

Is it possible to write a reproducer in Python?

The string returned by _PyUnicode_TransformDecimalAndSpaceToASCII() is still a NUL-terminated, but the NUL can be far past the end of the string.

msg321616 - (view)

Author: Jonas H. (jonash) *

Date: 2018-07-13 14:46

I don't think this can be tested with Python code, unless you can make sure the target buffer _PyUnicode_TransformDecimalAndSpaceToASCII operates on is initialised with garbage bytes.

msg321617 - (view)

Author: STINNER Victor (vstinner) * (Python committer)

Date: 2018-07-13 14:52

I don't think this can be tested with Python code, unless you can make sure the target buffer _PyUnicode_TransformDecimalAndSpaceToASCII operates on is initialised with garbage bytes.

We just have to call float() with a non-ASCII string, no? An assertion must fail without the fix when Python is compiled in debug mode.

msg321618 - (view)

Author: Jonas H. (jonash) *

Date: 2018-07-13 14:54

The assertion in the patched code, yes. The segfault in the unpatched code, no.

msg321621 - (view)

Author: Inada Naoki (methane) * (Python committer)

Date: 2018-07-13 15:34

When I commented out the fix:

$ ./python -c 'int("こんにちは")' python: Objects/unicodeobject.c:484: _PyUnicode_CheckConsistency: Assertion `PyUnicode_READ(kind, data, ascii->length) == 0' failed. Aborted

I added this to test cases.

msg321641 - (view)

Author: Inada Naoki (methane) * (Python committer)

Date: 2018-07-14 03:06

New changeset 16dfca4d829e45f36e71bf43f83226659ce49315 by INADA Naoki in branch 'master': bpo-34087: Fix buffer overflow in int(s) and similar functions (GH-8274) https://github.com/python/cpython/commit/16dfca4d829e45f36e71bf43f83226659ce49315

msg321642 - (view)

Author: miss-islington (miss-islington)

Date: 2018-07-14 03:58

New changeset c721472fb83d1f7c7606bcf33ba2d42d6127a764 by Miss Islington (bot) in branch '3.7': bpo-34087: Fix buffer overflow in int(s) and similar functions (GH-8274) https://github.com/python/cpython/commit/c721472fb83d1f7c7606bcf33ba2d42d6127a764

msg321645 - (view)

Author: Inada Naoki (methane) * (Python committer)

Date: 2018-07-14 07:38

New changeset b2f8aa0c998d331ab2b4c701756a6427c0e91d48 by INADA Naoki in branch '3.6': bpo-34087: Backport tests for int/float/complex (GH-8274) https://github.com/python/cpython/commit/b2f8aa0c998d331ab2b4c701756a6427c0e91d48

History

Date

User

Action

Args

2022-04-11 14:59:03

admin

set

github: 78268

2018-08-27 11:42:54

methane

link

issue34355 superseder

2018-07-14 07:38:21

methane

set

messages: +

2018-07-14 04:17:56

methane

set

pull_requests: + <pull%5Frequest7815>

2018-07-14 03:59:56

methane

set

status: open -> closed
stage: patch review -> resolved
resolution: fixed
versions: + Python 3.8

2018-07-14 03:58:16

miss-islington

set

nosy: + miss-islington
messages: +

2018-07-14 03:07:17

miss-islington

set

pull_requests: + <pull%5Frequest7814>

2018-07-14 03:06:52

methane

set

messages: +

2018-07-13 15:38:53

ezio.melotti

set

nosy: + ezio.melotti

2018-07-13 15:34:49

methane

set

messages: +

2018-07-13 14:54:19

jonash

set

messages: +

2018-07-13 14:52:34

vstinner

set

messages: +

2018-07-13 14:46:27

jonash

set

messages: +

2018-07-13 13:57:26

serhiy.storchaka

set

nosy: + serhiy.storchaka
messages: +

2018-07-13 13:12:17

methane

set

keywords: + patch
stage: patch review
pull_requests: + <pull%5Frequest7809>

2018-07-13 13:08:43

methane

set

title: float(unicode) may cause segmentation fault -> int(s), float(s) and others may cause segmentation fault

2018-07-13 13:00:52

vstinner

set

messages: +

2018-07-13 12:56:10

methane

set

messages: +
title: django: segmentation fault on random places -> float(unicode) may cause segmentation fault

2018-07-13 12:50:55

fenrrir

set

messages: +

2018-07-13 11:48:23

jonash

set

messages: +

2018-07-13 11:30:51

methane

set

messages: +

2018-07-13 11:14:43

methane

set

messages: +

2018-07-13 11:00:19

jonash

set

messages: +

2018-07-13 10:57:51

methane

set

messages: +

2018-07-13 10:42:08

vstinner

set

messages: +

2018-07-13 10:41:49

jonash

set

files: + testproj.tar.gz

messages: +

2018-07-13 09:43:16

methane

set

title: django: segmentation fault on garbage collection in visit_decref() -> django: segmentation fault on random places

2018-07-13 09:42:16

methane

set

messages: +

2018-07-12 22🔞13

methane

set

messages: +

2018-07-12 19:14:07

fenrrir

set

messages: +

2018-07-12 19:07:07

jonash

set

messages: +

2018-07-12 15:32:16

methane

set

messages: +

2018-07-12 15:31:37

fenrrir

set

messages: +

2018-07-12 14:54:47

jonash

set

messages: +

2018-07-12 14:54:04

jonash

set

nosy: + jonash
messages: +

2018-07-12 10:05:00

methane

set

messages: +

2018-07-11 18:02:20

fenrrir

set

messages: +

2018-07-11 14:17:01

vstinner

set

title: Segmentation fault on visit_decref -> django: segmentation fault on garbage collection in visit_decref()

2018-07-11 14:03:42

vstinner

set

nosy: + vstinner
messages: +

2018-07-11 13:27:13

fenrrir

set

messages: +

2018-07-11 03:00:02

methane

set

messages: +

2018-07-11 01:39:56

fenrrir

set

messages: +

2018-07-11 01:02:16

methane

set

nosy: + methane
messages: +

2018-07-10 19:36:06

fenrrir

create