(original) (raw)

> long as numpy.loadtxt is explicitly documented as only working with
\> latin-1 encoded files (it currently isn't), there's no problem.

Actually there is problem. If it explicitly specified the encoding as
latin-1 when opening the file then it could document the fact that it
works for latin-1 encoded files. However it actually uses the system
default encoding to read the file

and then converts the strings to
bytes with the as\_bytes function that is hard-coded to use latin-1:
https://github.com/numpy/numpy/blob/master/numpy/compat/py3k.py#L28

So it only works if the system default encoding is latin-1 and the
file content is white-space and newline compatible with latin-1.
Regardless of whether the file itself is in utf-8 or latin-1 it will
only work if the system default encoding is latin-1\. I've never used a
system that had latin-1 as the default encoding (unless you count
cp1252 as latin-1).

\> If it's supposed to work with other encodings (but the entire file is
\> still required to use a consistent encoding), then it just needs
\> encoding and errors arguments to fit the Python 3 text model (with
\> "latin-1" documented as the default encoding).

This is the right solution. Have an encoding argument, document the
fact that it will use the system default encoding if none is
specified, and re-encode using the same encoding to fit any dtype='S'
bytes column. This will then work for any encoding including the ones
that aren't ASCII-compatible (e.g. utf-16).

> default behaviour, since passing something like
\> codecs.getdecoder("utf-8") as a column converter should do the right
\> thing.

That's just getting silly IMO. If the file uses mixed encodings then I

don't consider it a valid "text file" and see no reason for loadtxt to
support reading it.