Unicode includes Line Separator U+2028 and Paragraph Separator U+2029 line ending characters. The readlines method of the file object returned by the built-in open does not treat these characters as line ends although the object returned by codecs.open(..., encoding='utf-8') does. The attached program creates a UTF-8 file containing three lines with the second line ended with a Paragraph Separator. The program then reads this file back in as a text file. Only two lines are seen when reading the file back in. The desired behaviour is for the file to be read in as three lines.
This seems to be because codecs.StreamReader.readlines() function does this: def readlines(self, sizehint=None, keepends=True): data = self.read() return data.splitlines(keepends) But the io readlines() functions make multiple calls to readline() instead. Here is the test case which passes on the codecs readlines() but fails on the io readlines().
By design, readlines() only recognizes those characters which are official line separators on various OSes (\n, \r, \r\n). This is important for proper parsing of log files, internet protocols, etc. If you want to split on all line separators recognized by the unicode spec, use str.splitlines().
