Issue 12512: codecs: StreamWriter issues with stateful codecs after a seek or with append mode (original) (raw)

Issue12512

Created on 2011-07-07 11:14 by vstinner, last changed 2022-04-11 14:57 by admin. This issue is now closed.

Messages (3)
msg139969 - (view)	Author: STINNER Victor (vstinner) *	Date: 2011-07-07 11:14
The following code fails with an AssertionError('###\ufeffdef'): import codecs _open = codecs.open #_open = open filename = "test" with _open(filename, 'w', encoding='utf_16') as f: f.write('abc') pos = f.tell() with _open(filename, 'w', encoding='utf_16') as f: f.seek(pos) f.write('def') f.seek(0) f.write('###') with _open(filename, 'r', encoding='utf_16') as f: content = f.read() assert content == '###def', ascii(content) It is a bug in StreamWriter.seek(): it should update the encoder state to not write a new BOM. It has to be fixed in the StreamWriter class of each stateful codec, or a stateful StreamWriter class should be implemented in the codecs module. Python supports the following stateful codecs: * cp932 * cp949 * cp950 * euc_jis_2004 * euc_jisx2003 * euc_jp * euc_kr * gb18030 * gbk * hz * iso2022_jp * iso2022_jp_1 * iso2022_jp_2 * iso2022_jp_2004 * iso2022_jp_3 * iso2022_jp_ext * iso2022_kr * shift_jis * shift_jis_2004 * shift_jisx0213 * utf_8_sig * utf_16 * utf_32 This bug has already been fixed in TextIOWrapper: issue #5006.
msg139970 - (view)	Author: STINNER Victor (vstinner) *	Date: 2011-07-07 11:20
There is a similar bug for append mode: import codecs _open = codecs.open #_open = open filename = "test" with _open(filename, 'w', encoding='utf_16') as f: f.write('abc') with _open(filename, 'a', encoding='utf_16') as f: f.write('def') with _open(filename, 'r', encoding='utf_16') as f: content = f.read() assert content == 'abcdef', ascii(content) This bug has also been fixed by the issue #5006 in the io module.
msg297127 - (view)	Author: STINNER Victor (vstinner) *	Date: 2017-06-28 01:39
Sorry, I just lost track of this issue and so will just close it.