[Python-Dev] Improve open() to support reading file starting with an unicode BOM (original) (raw)
Henning von Bargen henning.vonbargen at arcor.de
Sun Jan 10 12:10:02 CET 2010
- Previous message: [Python-Dev] Unladen cPickle speedups in 2.7 & 3.1
- Next message: [Python-Dev] Improve open() to support reading file starting with an unicode BOM
- Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]
If Python should support BOM when reading text files, it should also be able to write such files.
An encoding="BOM" argument wouldn't help here, because it does not specify which encoding to use actually: UFT-8, UTF-16-LE or what?
That would be a point against encoding="BOM" and pro an additional keyword argument "use_bom" or whatever with the following values:
None: default (old) behaviour: don't handle BOM at all
True: reading: expect BOM (raising an exception if it's missing). The encoding argument must be None or it must match the encoding implied by the BOM writing: write a BOM. The encoding argument must be one of the UTF encodings. False: reading: If a BOM is present, use it to determine the file encoding. The encoding argument must be None or it must match the encoding implied by the BOM. (*) Otherwise, use the encoding argument to determine the encoding. writing: do not write a BOM. Use the encoding argument.
(*) This is a question of taste. I think some people would prefer a fourth value "AUTO" instead, or to swap the behaviour of None and False.
Henning
P.S. To make things worse, I have sometimes seen XML files with a UTF-8 BOM, but an XML encoding declaration of "iso-8859-1". For such files, whatever you guess will be wrong anyway...
- Previous message: [Python-Dev] Unladen cPickle speedups in 2.7 & 3.1
- Next message: [Python-Dev] Improve open() to support reading file starting with an unicode BOM
- Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]