Issue 32395: asyncio.StreamReader.readuntil is not general enough (original) (raw)

Issue32395

Created on 2017-12-21 05:52 by Bruce Merry, last changed 2022-04-11 14:58 by admin. This issue is now closed.

Messages (4)
msg308852 - (view)	Author: Bruce Merry (Bruce Merry)	Date: 2017-12-21 05:52
I'd proposed one specific solution in Issue 32052 which asvetlov didn't like, so as requested I'm filing a bug about the problem rather than the solution. The specific case I have is reading a protocol in which either \r or \n can be used to terminate lines. With StreamReader.readuntil, it's only possible to specify one separator, so it can't easily be used (*). Some nice-to-have features, from specific to general: 1. Specify multiple alternate separators. 2. Specify a regex for a separator. 3. Specify a regex for the line. 4. Specify a callback that takes a string and returns the position of the end of the line, if any. Of course, some of these risk quadratic-time behaviour if they have to check the whole buffer every time the buffer is extended, so that would need to be considered in the design. In the last case, the callback could take care of it itself by maintaining internal state. (*) I actually have a solution for this case (https://github.com/ska-sa/aiokatcp/blob/bd8263cefe213003a218fac0dd8c5207cc76aeef/aiokatcp/connection.py#L44-L52), but it only works because \r and \n are semantically equivalent in the particular protocol I'm parsing.
msg308853 - (view)	Author: Andrew Svetlov (asvetlov) *	Date: 2017-12-21 06:06
Support multiple separators looks easy, I don't expect any performance impact. Like we already have it for strings: s.startswith(('\n', '\r')) Regexps are more expensive thing, callbacks are kind of evil. Let's add a patch for multiple separators first, maybe it covers 99.9% use cases.
msg352078 - (view)	Author: Karthikeyan Singaravelan (xtreak) *	Date: 2019-09-12 09:37
I think this is a duplicate of where multiple separators are requested for readuntil. I guess we can close one of them as duplicates.
msg352080 - (view)	Author: Andrew Svetlov (asvetlov) *	Date: 2019-09-12 09:39
Agree