The documentation for the xml.sax.handler.property_xml_string SAX property states that it should be "data type: String". However when retrieving this value in Python 3.1 it returns a bytes object instead. This makes handling the returned value very difficult because there is no method for retrieving the character set encoding that the XML was originally encoded with. This is currently blocking the port of SimpleTAL to Python 3 achieving feature parity with Python 2.
I'm not familiar with the inner workings of the expat integration with Python, so the attached patches need careful review. The first patch (expatreader.py.patch) is the minimum to resolve this issue. The second patch (expatreader.py.patch2) also exposes the version and encoding parameters via the Locator2 interface (http://www.saxproject.org/apidoc/org/xml/sax/ext/Locator2.html), which I'd recommend including.
See additional research and discussion in the comments of PR GH-9715. Simply changing this to return a string rather than bytes would break backwards compatibility. I certainly agree that this should have returned a string in the first place, especially since the Unicode decoding is otherwise completely abstracted away and the encoding used is not made available. Our options: 1. Return a string starting with 3.8, document the change in What's New & fix the docs for older 3.x. 2. Continue returning bytes, update the docs for all 3.x that this returns bytes, and that there's no good way to know the proper encoding to use for decoding it. 3. As 2 above, but also expose the encoding used. Since this appears to be rarely used and option 3 requires significantly more effort than the others, I am against it. Option 2 seems the safest, but I'd like to hear more from those more experienced with XML.
The other thing to consider which also supports option 2 is that xml.parsers.expat provides an interface to the Expat parser which is easier to use and more complete than the Sax parser implementation and is the implementation likely to be used by anyone needing a streaming parser.
