[Python-Dev] csv module and universal newlines (original) (raw)

Andrew McNamara andrewm at object-craft.com.au
Mon Jan 10 05:44:41 CET 2005

This item, from the TODO list, has been bugging me for a while:

* Reader and universal newlines don't interact well, reader doesn't honour Dialect's lineterminator setting. All outstanding bug id's (789519, 944890, 967934 and 1072404) are related to this - it's a difficult problem and further discussion is needed.

The csv parser consumes lines from an iterator, but it also has it's own idea of end-of-line conventions, which are currently only used by the writer, not the reader, which is a source of much confusion. The writer, by default, also attempts to emit a \r\n sequence, which results in more confusion unless the file is opened in binary mode.

I'm looking for suggestions for how we can mitigate these problems (without breaking things for existing users).

The standard file iterator includes the end-of-line characters in the returned string. One potentional solution is, then, to ignore the line chunking done by the file iterator, and logically concatenate the source lines until the csv parser's idea of lineterminator is seen - but this defeats negates the benefits of using an iterator.

Another option might be to provide a new interface that relies on a file-like object being supplied. The lineterminator character would only be used with this interface, with the current interface falling back to using only \n. Rather a drastic solution.

Any other ideas?

-- Andrew McNamara, Senior Developer, Object Craft http://www.object-craft.com.au/

More information about the Python-Dev mailing list