[Python-Dev] Python-3.0, unicode, and os.environ (original) (raw)

glyph at divmod.com glyph at divmod.com
Fri Dec 12 21:54:07 CET 2008

On 02:23 pm, curt at hagenlocher.org wrote:

On Fri, Dec 12, 2008 at 6:19 AM, Antoine Pitrou <solipsis at pitrou.net> wrote:

Curt Hagenlocher <curt hagenlocher.org> writes:

No, but it also has to interact with filesystems of possibly invalid or indeterminate encodings. What does java.io do? My point was that Python doesn't have to interact with the Java IO libraries, while it has to interact with the Unix and Windows IO APIs. Of course. But the Java IO libraries have to interact with the Unix and Windows IO APIs as well. It might be interesting to know how they handle similar situations.

Apparently Java has the facilities to do the right thing, but actually it's just broken.

My locale says UTF-8. However, if I create a non-decodable file with Python (2), there are three ways I can tell Java to open it: I can ask for it with a string (that won't work, because no valid UTF-8 string maps to an undecodable string, pretty much by definition). I can list the directory that it's in (presuming that that's a directory) and get a java.io.File, which could be retaining all the interesting information, or I can use a URI, which is a string that resolves to octets before it resolves to characters again.

However, it looks like Java screws up in every case.

Here's a transcript from the ever-helpful jython:

glyph at nhuvasarim:~/tmp$ python Python 2.5.2 (r252:60911, Jul 31 2008, 17:28:52) [GCC 4.2.3 (Ubuntu 4.2.3-2ubuntu7)] on linux2 Type "help", "copyright", "credits" or "license" for more information.

file("\xff\xff", "wb").write("lolz\n") glyph at nhuvasarim:~/tmp$ jython Jython 2.2.1 on java1.6.0_07 Type "copyright", "credits" or "license" for more information. from java.io import File fileList = File(".").listFiles() fileList array(java.io.File,[./ fileList[0].class <jclass java.io.File 1> from java.io import FileReader FileReader(fileList[0]) Traceback (innermost last): File "", line 1, in ? at java.io.FileInputStream.open(Native Method) at java.io.FileInputStream.(FileInputStream.java:106) at java.io.FileReader.(FileReader.java:55) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27) at java.lang.reflect.Constructor.newInstance(Constructor.java:513)

java.io.FileNotFoundException: java.io.FileNotFoundException: ./ÿFDÿFD (No such file or directory)

from java.net import URI u = URI("file:///home/glyph/tmp/%ff%ff") FileReader(File(u)) Traceback (innermost last): File "", line 1, in ? at java.io.FileInputStream.open(Native Method) at java.io.FileInputStream.(FileInputStream.java:106) at java.io.FileReader.(FileReader.java:55) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27) at java.lang.reflect.Constructor.newInstance(Constructor.java:513)

java.io.FileNotFoundException: java.io.FileNotFoundException: /home/glyph/tmp/ÿFDÿFD (No such file or directory)

More information about the Python-Dev mailing list