[Python-Dev] [Fwd: PEP 0305 (small problem with the CSV reader)] (original) (raw)

Andrew McNamara andrewm at object-craft.com.au
Wed Mar 28 02:44:55 CEST 2007

First of all, let me say thank you for the CSV module.

Thanks.

I've been using it and today is the first time I hit a minor bump in the road. What happened is I opened this file with genome annotations with a long field and the error "field larger than field limit" showed up. From what I can see it is in the "static int parseaddchar(ReaderObj *self, char c)" function. This function uses the static long fieldlimit = 128 * 1024; /* max parsed field size */ I'm not sure if this is supposed to be recomputed or if there is something I need to do to change it, but for right now it just says my row is bigger than 131,072 and stops. I don't think Python 2.5 has any such string length limitations and this shouldn't be.

This limit was added back in January 2005 to provide some protection against the situation where the parser is returning fields directly from a file, and the file contains a mismatched quote character: this would otherwise result in the entire file being unexpectedly read into memory.

You can change the limit with the csv.field_size_limit() method. As you note, it defaults to 128K, but you can set it to anything up to (2**31)-1 or 2147483647 (about 2 billion).

BTW, I've taken the liberty of CC'ing this to the python-dev list, so the motivation for this feature is recorded - it caused me some head scratching, and I added it.

-- Andrew McNamara, Senior Developer, Object Craft http://www.object-craft.com.au/

More information about the Python-Dev mailing list