[Python-Dev] Decoding incomplete unicode (original) (raw)

M.-A. Lemburg mal at egenix.com
Wed Aug 18 00:25:21 CEST 2004

Walter Dörwald wrote:

M.-A. Lemburg wrote:

Overall, I don't like the idea of adding extra APIs breaking the existing codec API. Adding a final argument that defaults to False doesn't break the API for the callers, only for the implementors. And if we drop the final argument, the API is completely backwards compatible both for users and implementors. The only thing that gets added is the feed() method, that implementors don't have to overwrite.

I believe that we can extend stream codecs to also work in a feed mode without breaking the API. Abandoning the final argument and adding a feed() method would IMHO be the simplest way to do this. But then there's no way to make sure that every byte from the input stream is really consumed.

I've thought about this some more. Perhaps I'm still missing something, but wouldn't it be possible to add a feeding mode to the existing stream codecs by creating a new queue data type (much like the queue you have in the test cases of your patch) and using the stream codecs on these ?

I think such a queue would be generally useful in other contexts as well, e.g. for implementing fast character based pipes between threads, non-Unicode feeding parsers, etc. Using such a type you could potentially add a feeding mode to stream or file-object API based algorithms very easily.

We could then have a new class, e.g. FeedReader, which wraps the above in a nice API, much like StreamReaderWriter and StreamRecoder.

-- Marc-Andre Lemburg eGenix.com

Professional Python Services directly from the Source (#1, Aug 17 2004)

Python/Zope Consulting and Support ... http://www.egenix.com/ mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/

::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! ::::

More information about the Python-Dev mailing list