Issue 17898: gettext bug while parsing plural-forms metadata (original) (raw)
Created on 2013-05-03 18:19 by straz, last changed 2022-04-11 14:57 by admin. This issue is now closed.
Messages (11)
Author: Steve Strassmann (straz)
Date: 2013-05-03 18:19
The gettext.py parser used by django (lib/python2.7/gettext.py), GNUTranslations._parse(), around line 313 does not use clean values for k,v on each iteration ("for item in tmsg.splitlines():")
To reproduce the problem (see traceback, below), try parsing a .PO file containing two headers like this, with a comment header immediately following a plurals header. This example was created by calling msgcat to combine several .po files into a single .po file. Msgcat inserted the comment line.
"Plural-Forms: nplurals=2; plural=(n != 1);\n" "#-#-#-#-# messages.po (EdX Studio) #-#-#-#-#\n"
Parsing the first header binds the inner loop variables: k= plural-forms v= ['nplurals=2', ' plural=(n != 1)', '']
Parsing the second header leaves k,v untouched, which then causes an improper attempt to parse (since it's a comment, no further parsing of k,v should occur) v = v.split(';')
Bug workaround: I use polib to read and immediately save the file. This reorders the metadata to avoid presenting the parser with something that will break it.
Recommended bug fix: on each iteration over tmsg.splitlines, reset the values of k,v = (None, None)
Traceback: File "/Users/sstrassmann/src/mitx_all/python/lib/python2.7/site-packages/django/core/handlers/base.py" in get_response 89. response = middleware_method(request) File "/Users/sstrassmann/src/mitx_all/python/lib/python2.7/site-packages/django/middleware/locale.py" in process_request 24. translation.activate(language) File "/Users/sstrassmann/src/mitx_all/python/lib/python2.7/site-packages/django/utils/translation/init.py" in activate 105. return _trans.activate(language) File "/Users/sstrassmann/src/mitx_all/python/lib/python2.7/site-packages/django/utils/translation/trans_real.py" in activate 201. _active.value = translation(language) File "/Users/sstrassmann/src/mitx_all/python/lib/python2.7/site-packages/django/utils/translation/trans_real.py" in translation 191. current_translation = _fetch(language, fallback=default_translation) File "/Users/sstrassmann/src/mitx_all/python/lib/python2.7/site-packages/django/utils/translation/trans_real.py" in _fetch 180. res = _merge(localepath) File "/Users/sstrassmann/src/mitx_all/python/lib/python2.7/site-packages/django/utils/translation/trans_real.py" in _merge 156. t = _translation(path) File "/Users/sstrassmann/src/mitx_all/python/lib/python2.7/site-packages/django/utils/translation/trans_real.py" in _translation 138. t = gettext_module.translation('django', path, [loc], DjangoTranslation) File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/gettext.py" in translation 480. t = translations.setdefault(key, class(fp)) File "/Users/sstrassmann/src/mitx_all/python/lib/python2.7/site-packages/django/utils/translation/trans_real.py" in init 76. gettext_module.GNUTranslations.init(self, *args, **kw) File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/gettext.py" in init 180. self._parse(fp) File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/gettext.py" in _parse 315. v = v.split(';')
Exception Type: AttributeError at / Exception Value: 'list' object has no attribute 'split'
Author: R. David Murray (r.david.murray) * 
Date: 2013-05-03 19:27
Does this bear any relationship to issue 1475523? (And yes, I know it is...sad...that that issue hasn't been fixed yet.)
Author: Steve Strassmann (straz)
Date: 2013-05-03 19:38
There seem to be several bugs involving this particular inner loop in gettext._parse(), but I don't think they're equivalent.
The present bug () is that parsing a plural header breaks the following header when it happens to be a comment.
seems to involve multi-line handling
seems to involve breaking when the plural-forms value is empty.
Perhaps a useful design pattern to follow for code which executes this inner loop would be to have some initialization and loop invariants which are asserted true on each iteration. For example, properly initializing k and v on each iteration.
Author: Terry J. Reedy (terry.reedy) *
Date: 2013-05-10 19:13
Since the other two listed inner-loop issues apply to 3.x, I would guess that this does also. Steve, can you test with 3.3? And provide a minimal test case?
Author: Steve Strassmann (straz)
Date: 2013-05-10 20:07
Sorry, I haven't installed python 3.*, I just have default Mac OS python 2.7.
Here's a minimal test case. Tar expands to
file structure:
./test.py
./en/LC_MESSAGES/messages.po
$ ./test.py
Traceback (most recent call last):
File "./test.py", line 28, in
test()
File "./test.py", line 23, in test
gettext.install('messages', localedir=LOCALEDIR)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/gettext.py", line 494, in install
t = translation(domain, localedir, fallback=True, codeset=codeset)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/gettext.py", line 479, in translat
ion
t = translations.setdefault(key, class(fp))
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/gettext.py", line 180, in init
self._parse(fp)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/gettext.py", line 314, in _parse
v = v.split(';')
AttributeError: 'list' object has no attribute 'split'
Author: A.M. Kuchling (akuchling) *
Date: 2015-04-13 14:32
Proposed patch against 3.5.
Author: A.M. Kuchling (akuchling) *
Date: 2015-04-13 15:49
Adding a link to a bitbucket repo.
Author: A.M. Kuchling (akuchling) *
Date: 2015-04-13 15:55
I would apply this change to 3.4 and 3.5. Should I also backport it to 2.7? I think the same bug applies there, though I haven't verified this or tried my patch.
Author: Ned Deily (ned.deily) *
Date: 2015-04-13 17:30
LGTM. I think it should be backported to 2.7.
Author: Roundup Robot (python-dev) 
Date: 2015-04-14 14:19
New changeset c3d269c01671 by Andrew Kuchling in branch '2.7': #17898: reset k and v so that the loop doesn't use an old value https://hg.python.org/cpython/rev/c3d269c01671
Author: Roundup Robot (python-dev)
Date: 2015-04-14 14:25
New changeset 54df02192bfc by Andrew Kuchling in branch '3.4': #17898: reset k and v so that the loop doesn't use an old value https://hg.python.org/cpython/rev/54df02192bfc
History
Date
User
Action
Args
2022-04-11 14:57:45
admin
set
github: 62098
2015-04-14 14:37:15
akuchling
set
status: open -> closed
assignee: akuchling
resolution: fixed
stage: patch review -> resolved
2015-04-14 14:25:35
python-dev
set
messages: +
2015-04-14 14:19:09
python-dev
set
nosy: + python-dev
messages: +
2015-04-13 17:30:53
ned.deily
set
nosy: + ned.deily
messages: +
versions: + Python 3.4, Python 3.5
2015-04-13 15:55:03
akuchling
set
messages: +
2015-04-13 15:49:09
akuchling
set
hgrepos: + hgrepo304
messages: +
2015-04-13 14:32:25
akuchling
set
files: + issue17898.patch
nosy: + akuchling
messages: +
keywords: + patch
stage: test needed -> patch review
2013-05-10 20:07:10
straz
set
files: + pybug.tar.gz
messages: +
2013-05-10 19:13:00
terry.reedy
set
nosy: + terry.reedy
messages: +
stage: test needed
2013-05-03 19:38:03
straz
set
messages: +
2013-05-03 19:27:09
r.david.murray
set
nosy: + r.david.murray
messages: +
2013-05-03 18:40:33
nedbat
set
nosy: + nedbat
2013-05-03 18:19:46
straz
create