[Python-Dev] urllib.quote and unquote - Unicode issues (original) (raw)

M.-A. Lemburg mal at egenix.com
Wed Aug 6 19:34:48 CEST 2008

On 2008-08-06 18:55, Antoine Pitrou wrote:

Martin v. Löwis <martin v.loewis.de> writes:

URLs are just not made for non-ASCII characters. Perhaps they are not, but every non-English wiki (just to take a simple, generic example) potentially contains non-ASCII URLs. e.g. http://fr.wikipedia.org/wiki/%C3%89l%C3%A9phant http://wiki.python.org/moin/J%C3%BCrgenHermann (notice the utf-8 encoding in both)

Implement IRIs if you want non-ASCII characters; the rules are much clearer for these. I think most people would expect something which works with the current World Wide Web rather than a rigorous implementation of a specific RFC. Implementing RFCs is fine but it does not magically eliminate all problems, especially when the RFCs themselves are not in sync with real-world usage.

+1. Practicality beats purity...

The web is moving towards UTF-8 as standard Unicode encoding, so it's probably wise to follow that approach for quote().

http://en.wikipedia.org/wiki/Percent-encoding

The other way around will also have to deal with old-style URLs which typically still use the Latin-1 encoding which was the basis for HTML:

http://www.w3schools.com/TAGS/ref_urlencode.asp

So unquote() should probably try to decode using UTF-8 first and then fall back to Latin-1 if that doesn't work.

Whether the result of quote()/unquote() should be bytes or Unicode is a different story and probably also depends on what the application does with the result. I don't think there's a good general answer for that one, except maybe just going for one possible combination and document that.

-- Marc-Andre Lemburg eGenix.com

Professional Python Services directly from the Source (#1, Aug 05 2008)

Python/Zope Consulting and Support ... http://www.egenix.com/ mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/

:::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,MacOSX for free ! ::::

eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
        Registered at Amtsgericht Duesseldorf: HRB 46611

More information about the Python-Dev mailing list