Codereview request for 7096080: UTF8 update and new CESU-8 charset (original) (raw)

Xueming Shen xueming.shen at oracle.com
Wed Sep 28 19🔞34 UTC 2011

Hi,

[I combined the proposed charge for #7082884, in which no one appears to be interested:-) into this one]

Unicode Standard added "Addition Constraints on conversion of ill-formed UTF-8" in version 5.1 [1] and updated in 6.0 again with further "clarification" [2] regarding how a "conformance" implementation should handle ill-formed UTF-8 byte sequence. Basically it says

(1) the conversion process should not interpret any ill-formed code unit sequence (2) such process must not treat any adjacent well-formed code unit sequences as being part of those ill-formed code unit sequences (3) and recommend a "best practice" of "maximal valid sub-part" for replacement

The new UTF-8 charset implementation we put in JDK7 (and back-ported to previous release since then) follows the new constraints in most cases, except

(1) The decoder still accepts "historical" 3 bytes surrogates and 6 bytes surrogate pair (the encoder never outputs such sequence). Unicode Standard "tightened" its UTF-8 definition in ver 3.2 [3], as

 "Most notable among the corrigenda to the Standard is a further

tightening of the definition of UTF-8, to eliminate irregular UTF-8 and to bring the Unicode specification of UTF-8 more completely into line with other specifications of UTF-8."

So the 3-byte/6-byte surrogates are now defined as "ill-formed" code unit sequence, instead of "irregular" [5] in ver 3.1

(2) While no longer accepting the "historical" 5-byte, 6-byte UTF-8 byte sequence, the decoder treats these 5/6-byte sequence as ONE malformed unit. As a result these bytes get replaced by one replacement character, when "replace for malformed" is desirable (as in new String(bytes), for example). According the latest Unicode standard [2], however, because the leading byte of these 5/6-byte sequence is no longer an illegal appearance of the UTF-8, these bytes should be treated as 5-6 individual ill-formed bytes.

(3)Corner case like ill-formed byte sequence ED 31 is not handled correctly/ consistently, as described in #7082884 [6]

The reason behind (1) and (2) is mostly the compatibility concern. As suggested in TR#26 [4] (in which it defines CESU-8, a separate UTF encoding scheme that uses 3-6-byte sequence for supplementary characters, instead of 4-byte sequence in UTF-8), there are apps/data over there that do use surrogates pair in "UTF-8" form. To change the UTF-8 charset to follow standard obviously will break someone's code when they migrate/upgrade from JDK/JRE N to N+1, something we try really hard to avoid.

That said, given almost decade has passed and we are now at Unicode 6, I think the possibility of breaking someone's code/date of upgrading UTF-8 to do the "right thing" is small/minor. So I proposed here

(1) to upgrade the JDK8 UTF-8 implementation to strictly follow the standard to a) reject 3-byte surrogate/6-byte surrogate pair b) treats 5/6-byte surrogate as individual ill-formed bytes. c) fix the corner case bug #7082884 (2) to add CESU-8 charset into JDK/JRE's charset repository (for those still prefer/work on 3-6 bytes surrogate, in "UTF-8" form)

Here is the webrev. The change will need to go through some "in-compatible change" review process, but I think we can start the code review/discussion here first.

http://cr.openjdk.java.net/~sherman/7096080/webrev/ <http://cr.openjdk.java.net/%7Esherman/7096080/webrev/>

-Sherman

[1] http://www.unicode.org/versions/Unicode5.1.0/#Notable_Changes [2] http://www.unicode.org/versions/Unicode6.0.0/#Conformance_Changes [3] http://www.unicode.org/reports/tr28/tr28-3.html [4] http://unicode.org/reports/tr26/ [5] http://unicode.org/versions/corrigendum1.html [6] http://mail.openjdk.java.net/pipermail/core-libs-dev/2011-September/007722.html

More information about the core-libs-dev mailing list