🔫 Unicode Support Shootout

👍 The Good, the Bad, & the (mostly) Ugly 👎

‌

Tom Christiansen
<tchrist@perl.com>

These slides are at http://training.perl.com/OSCON2011/index.html

What is this?

• Get the current version of these slides in HTML, PDF, or POD from here:

   http://training.perl.com/OSCON2011/index.html 

• See Appendix 1: Fonts at the end of this talk for instructions on font set-up.
• These slides were written in pod using vi, with the help of a score of Unicode tools available via the link in Appendix 2: Tools.
• This slide show was built using Pod::S5 by Tom Linden, which in turn uses S5 by Eric Meyer. Slideshow controls appears if you hover near the bottom right corner, but I find it easier to use keystroke commands to navigate.
• I’ve used next to no HTML tricks here; nearly all fancy‐qua‐weird stuff you see here is actually Unicode.

🔎 Scope of This Talk

• I’ll first motivate you by quickly showing the sorts of Unicode you now routinely encounter in modern, real‐world text corpora.
• Then I’ll show you a pair of charts that give high‐level summaries of how far each of the more popular programming languages respectively goes — or fails to go — in its native, out‐of‐the‐box Unicode support.
• I’ll then look into each of the covered languages’ Unicode strengths and weaknesses.
• The particular languages I’ll be looking at are Javascript, PHP, Go, Python, Ruby, Java, and Perl.
• I’ve worked with Unicode daily for years, but I don’t know everything about everything. Languages are listed in increasing order of my own experience with them — so I may have missed stuff, especially in the earlier ones.

The Unicode Standard

• Because Unicode is a 📖 standard, get used to checking various components of The Unicode Standard, which at last glance comprised 11 standard annexes, 8 technical standards, and 10 technical reports. Here are some that stand out:
  • 📖 UAX #44: Unicode Character Database (includes all properties)
  • 📖 UTS #18: Unicode Regular Expressions (includes three requirement levels)
  • 📖 UAX #15: Unicode Normalization Forms (NFD, NFC, &c)
  • 📖 UTS #10: Unicode Collation Algorithm (UCA)
  • 📖 UAX #29: Unicode Text Segmentation (includes graphemes)
  • 📖 UAX #14: Unicode Line Breaking Algorithm
  • 📖 UAX #11: East Asian Width
• “A Unicode Technical Report (UTR) contains informative material. Conformance to the Unicode Standard does not imply conformance to any UTR. Other specifications, however, are free to make normative reference.”

A Sample Unicode Corpus

The PubMed Central Open Access corpus comprises around 11G of Unicode data spread across around 200k different files. Here are its most commonly occurring non‐ASCII code points:

   1  18.55%  18.55%  U+02013 ‹–›  GC=Pd    EN DASH                               31   0.49%  88.28%  U+02264 ‹≤›  GC=Sm    LESS-THAN OR EQUAL TO
   2   7.42%  25.97%  U+000A0  ⧆   GC=Zs    NO-BREAK SPACE                        32   0.44%  88.72%  U+000AE ‹®›  GC=So    REGISTERED SIGN
   3   7.03%  33.01%  U+000B1 ‹±›  GC=Sm    PLUS-MINUS SIGN                       33   0.43%  89.15%  U+000E4 ‹ä›  GC=Ll    LATIN SMALL LETTER A WITH DIAERESIS
   4   5.46%  38.47%  U+02212 ‹−›  GC=Sm    MINUS SIGN                            34   0.42%  89.57%  U+02020 ‹†›  GC=Po    DAGGER
   5   4.20%  42.66%  U+02003  ⧆   GC=Zs    EM SPACE                              35   0.41%  89.98%  U+003B4 ‹δ›  GC=Ll    GREEK SMALL LETTER DELTA
   6   3.68%  46.35%  U+003BC ‹μ›  GC=Ll    GREEK SMALL LETTER MU                 36   0.37%  90.35%  U+000E1 ‹á›  GC=Ll    LATIN SMALL LETTER A WITH ACUTE
   7   3.62%  49.97%  U+003B2 ‹β›  GC=Ll    GREEK SMALL LETTER BETA               37   0.34%  90.69%  U+02192 ‹→›  GC=Sm    RIGHTWARDS ARROW
   8   3.57%  53.53%  U+003B1 ‹α›  GC=Ll    GREEK SMALL LETTER ALPHA              38   0.33%  91.02%  U+000ED ‹í›  GC=Ll    LATIN SMALL LETTER I WITH ACUTE
   9   3.43%  56.96%  U+0200A  ⧆   GC=Zs    HAIR SPACE                            39   0.31%  91.33%  U+003C3 ‹σ›  GC=Ll    GREEK SMALL LETTER SIGMA
  10   3.22%  60.18%  U+000B0 ‹°›  GC=So    DEGREE SIGN                           40   0.30%  91.62%  U+000C5 ‹Å›  GC=Lu    LATIN CAPITAL LETTER A WITH RING ABOVE
  11   2.93%  63.11%  U+02009  ⧆   GC=Zs    THIN SPACE                            41   0.29%  91.92%  U+003BB ‹λ›  GC=Ll    GREEK SMALL LETTER LAMDA
  12   2.62%  65.73%  U+02019 ‹’›  GC=Pf    RIGHT SINGLE QUOTATION MARK           42   0.29%  92.21%  U+000F3 ‹ó›  GC=Ll    LATIN SMALL LETTER O WITH ACUTE
  13   2.51%  68.24%  U+02032 ‹′›  GC=Po    PRIME                                 43   0.26%  92.47%  U+02122 ‹™›  GC=So    TRADE MARK SIGN
  14   2.44%  70.68%  U+000D7 ‹×›  GC=Sm    MULTIPLICATION SIGN                   44   0.26%  92.73%  U+02236 ‹∶›  GC=Sm    RATIO
  15   2.04%  72.72%  U+0201D ‹”›  GC=Pf    RIGHT DOUBLE QUOTATION MARK           45   0.22%  92.95%  U+003C7 ‹χ›  GC=Ll    GREEK SMALL LETTER CHI
  16   2.04%  74.76%  U+0201C ‹“›  GC=Pi    LEFT DOUBLE QUOTATION MARK            46   0.21%  93.17%  U+02021 ‹‡›  GC=Po    DOUBLE DAGGER
  17   1.54%  76.30%  U+00394 ‹Δ›  GC=Lu    GREEK CAPITAL LETTER DELTA            47   0.21%  93.37%  U+003C4 ‹τ›  GC=Ll    GREEK SMALL LETTER TAU
  18   1.42%  77.71%  U+000B5 ‹µ›  GC=Ll    MICRO SIGN                            48   0.20%  93.57%  U+003B8 ‹θ›  GC=Ll    GREEK SMALL LETTER THETA
  19   1.34%  79.05%  U+003B3 ‹γ›  GC=Ll    GREEK SMALL LETTER GAMMA              49   0.20%  93.77%  U+003B5 ‹ε›  GC=Ll    GREEK SMALL LETTER EPSILON
  20   1.21%  80.26%  U+000E9 ‹é›  GC=Ll    LATIN SMALL LETTER E WITH ACUTE       50   0.18%  93.95%  U+02026 ‹…›  GC=Po    HORIZONTAL ELLIPSIS
  21   1.15%  81.41%  U+02014 ‹—›  GC=Pd    EM DASH                               51   0.16%  94.11%  U+02211 ‹∑›  GC=Sm    N-ARY SUMMATION
  22   1.14%  82.55%  U+02018 ‹‘›  GC=Pi    LEFT SINGLE QUOTATION MARK            52   0.15%  94.26%  U+003C0 ‹π›  GC=Ll    GREEK SMALL LETTER PI
  23   1.00%  83.54%  U+000A9 ‹©›  GC=So    COPYRIGHT SIGN                        53   0.15%  94.40%  U+000EF ‹ï›  GC=Ll    LATIN SMALL LETTER I WITH DIAERESIS
  24   0.71%  84.25%  U+02265 ‹≥›  GC=Sm    GREATER-THAN OR EQUAL TO              54   0.15%  94.55%  U+000A7 ‹§›  GC=So    SECTION SIGN
  25   0.60%  84.85%  U+000F6 ‹ö›  GC=Ll    LATIN SMALL LETTER O WITH DIAERESIS   55   0.15%  94.70%  U+02005  ⧆   GC=Zs    FOUR-PER-EM SPACE
  26   0.60%  85.45%  U+000B7 ‹·›  GC=Po    MIDDLE DOT                            56   0.14%  94.84%  U+003C9 ‹ω›  GC=Ll    GREEK SMALL LETTER OMEGA
  27   0.60%  86.05%  U+02022 ‹•›  GC=Po    BULLET                                57   0.14%  94.98%  U+000E8 ‹è›  GC=Ll    LATIN SMALL LETTER E WITH GRAVE
  28   0.59%  86.64%  U+0223C ‹∼›  GC=Sm    TILDE OPERATOR                        58   0.13%  95.12%  U+000F8 ‹ø›  GC=Ll    LATIN SMALL LETTER O WITH STROKE
  29   0.57%  87.22%  U+003BA ‹κ›  GC=Ll    GREEK SMALL LETTER KAPPA              59   0.13%  95.24%  U+003C1 ‹ρ›  GC=Ll    GREEK SMALL LETTER RHO
  30   0.57%  87.79%  U+000FC ‹ü›  GC=Ll    LATIN SMALL LETTER U WITH DIAERESIS   60   0.13%  95.37%  U+000E3 ‹ã›  GC=Ll    LATIN SMALL LETTER A WITH TILDE 

PMC: Astral Projections

Here are that corpus’s code points above the BMP, in order of popularity. Note 👽 Area 51 !

  1  U+01D49E ‹𝒞›  GC=Lu    MATHEMATICAL SCRIPT CAPITAL C        31  U+01D4C2 ‹𝓂›  GC=Ll    MATHEMATICAL SCRIPT SMALL M
  2  U+01D4AF ‹𝒯›  GC=Lu    MATHEMATICAL SCRIPT CAPITAL T        32  U+01D54D ‹𝕍›  GC=Lu    MATHEMATICAL DOUBLE-STRUCK CAPITAL V
  3  U+01D4AE ‹𝒮›  GC=Lu    MATHEMATICAL SCRIPT CAPITAL S        33  U+01D4B6 ‹𝒶›  GC=Ll    MATHEMATICAL SCRIPT SMALL A
  4  U+01D49F ‹𝒟›  GC=Lu    MATHEMATICAL SCRIPT CAPITAL D        34  U+01D4BE ‹𝒾›  GC=Ll    MATHEMATICAL SCRIPT SMALL I
  5  U+01D4B3 ‹𝒳›  GC=Lu    MATHEMATICAL SCRIPT CAPITAL X        35  U+01D4CC ‹𝓌›  GC=Ll    MATHEMATICAL SCRIPT SMALL W
  6  U+01D4A9 ‹𝒩›  GC=Lu    MATHEMATICAL SCRIPT CAPITAL N        36  U+01D516 ‹𝔖›  GC=Lu    MATHEMATICAL FRAKTUR CAPITAL S
  7  U+01D4AB ‹𝒫›  GC=Lu    MATHEMATICAL SCRIPT CAPITAL P        37  U+01D4CF ‹𝓏›  GC=Ll    MATHEMATICAL SCRIPT SMALL Z
  8  U+01D4A2 ‹𝒢›  GC=Lu    MATHEMATICAL SCRIPT CAPITAL G        38  U+01D53B ‹𝔻›  GC=Lu    MATHEMATICAL DOUBLE-STRUCK CAPITAL D
  9  U+01D49C ‹𝒜›  GC=Lu    MATHEMATICAL SCRIPT CAPITAL A        39  U+01D54B ‹𝕋›  GC=Lu    MATHEMATICAL DOUBLE-STRUCK CAPITAL T
 10  U+01D53C ‹𝔼›  GC=Lu    MATHEMATICAL DOUBLE-STRUCK CAPITAL E 40  U+01D4BB ‹𝒻›  GC=Ll    MATHEMATICAL SCRIPT SMALL F
 11  U+01D4AA ‹𝒪›  GC=Lu    MATHEMATICAL SCRIPT CAPITAL O        41  U+01D4CA ‹𝓊›  GC=Ll    MATHEMATICAL SCRIPT SMALL U
 12  U+01D4A5 ‹𝒥›  GC=Lu    MATHEMATICAL SCRIPT CAPITAL J        42  U+01D507 ‹𝔇›  GC=Lu    MATHEMATICAL FRAKTUR CAPITAL D
 13  U+01D4A6 ‹𝒦›  GC=Lu    MATHEMATICAL SCRIPT CAPITAL K        43  U+01D542 ‹𝕂›  GC=Lu    MATHEMATICAL DOUBLE-STRUCK CAPITAL K
 14  U+01D4B1 ‹𝒱›  GC=Lu    MATHEMATICAL SCRIPT CAPITAL V        44  U+01D546 ‹𝕆›  GC=Lu    MATHEMATICAL DOUBLE-STRUCK CAPITAL O
 15  U+01D4B2 ‹𝒲›  GC=Lu    MATHEMATICAL SCRIPT CAPITAL W        45  U+01D4BD ‹𝒽›  GC=Ll    MATHEMATICAL SCRIPT SMALL H
 16  U+01D4B4 ‹𝒴›  GC=Lu    MATHEMATICAL SCRIPT CAPITAL Y        46  U+01D4C5 ‹𝓅›  GC=Ll    MATHEMATICAL SCRIPT SMALL P
 17  U+01D4B5 ‹𝒵›  GC=Lu    MATHEMATICAL SCRIPT CAPITAL Z        47  U+01D505 ‹𝔅›  GC=Lu    MATHEMATICAL FRAKTUR CAPITAL B
 18  U+01D4B0 ‹𝒰›  GC=Lu    MATHEMATICAL SCRIPT CAPITAL U        48  U+01D50E ‹𝔎›  GC=Lu    MATHEMATICAL FRAKTUR CAPITAL K
 19  U+01D4AC ‹𝒬›  GC=Lu    MATHEMATICAL SCRIPT CAPITAL Q        49  U+01D541 ‹𝕁›  GC=Lu    MATHEMATICAL DOUBLE-STRUCK CAPITAL J
 20  U+01D54A ‹𝕊›  GC=Lu    MATHEMATICAL DOUBLE-STRUCK CAPITAL S 50  U+01D543 ‹𝕃›  GC=Lu    MATHEMATICAL DOUBLE-STRUCK CAPITAL L
 21  U+01D539 ‹𝔹›  GC=Lu    MATHEMATICAL DOUBLE-STRUCK CAPITAL B 51  U+100002  ⧆   GC=Co    <private use code point in plane 16>
 22  U+01D5A7 ‹𝖧›  GC=Lu    MATHEMATICAL SANS-SERIF CAPITAL H    52  U+01D4B8 ‹𝒸›  GC=Ll    MATHEMATICAL SCRIPT SMALL C
 23  U+01D517 ‹𝔗›  GC=Lu    MATHEMATICAL FRAKTUR CAPITAL T       53  U+01D4C1 ‹𝓁›  GC=Ll    MATHEMATICAL SCRIPT SMALL L
 24  U+01D4C3 ‹𝓃›  GC=Ll    MATHEMATICAL SCRIPT SMALL N          54  U+01D53D ‹𝔽›  GC=Lu    MATHEMATICAL DOUBLE-STRUCK CAPITAL F
 25  U+01D535 ‹𝔵›  GC=Ll    MATHEMATICAL FRAKTUR SMALL X         55  U+01D53E ‹𝔾›  GC=Lu    MATHEMATICAL DOUBLE-STRUCK CAPITAL G
 26  U+01D4BF ‹𝒿›  GC=Ll    MATHEMATICAL SCRIPT SMALL J          56  U+01D54C ‹𝕌›  GC=Lu    MATHEMATICAL DOUBLE-STRUCK CAPITAL U
 27  U+01D540 ‹𝕀›  GC=Lu    MATHEMATICAL DOUBLE-STRUCK CAPITAL I 57  U+01D6A4 ‹𝚤›  GC=Ll    MATHEMATICAL ITALIC SMALL DOTLESS I
 28  U+01D465 ‹𝑥›  GC=Ll    MATHEMATICAL ITALIC SMALL X          58  U+01D7D9 ‹𝟙›  GC=Nd    MATHEMATICAL DOUBLE-STRUCK DIGIT ONE
 29  U+01D4CE ‹𝓎›  GC=Ll    MATHEMATICAL SCRIPT SMALL Y
 30  U+01D538 ‹𝔸›  GC=Lu    MATHEMATICAL DOUBLE-STRUCK CAPITAL A 

PMC: Graphemes & Ideographs

• PMC has plenty of “extended grapheme clusters”, glyphs that occupy more than one code point. You must make sure never to split up graphemes. Code points with the property Bidi_Class=Nonspacing_Mark (BC=NSM for short) are here in abundance: ◌̂, ◌̅, ◌̄, ◌̇, ◌̃, ◌⃗, ◌⃞, ◌̲, ◌̀, ◌͂, ◌́, ◌̊, ◌̆, ◌︀, ◌̵, ◌̑, ◌̧, ◌⃑, ◌̣, ◌̈, ◌̣, ◌̷, ◌́, ◌̋, ◌̌, ◌͘, ◌⃛, ◌֗, ◌̨, ◌̓, ◌̈́, ◌ْ, and ◌ྞ.
• You get to deal with CJK unified ideographs like 所 文 有 條 獨 祴 舰 蜓 鞒 鞔 㛧 㛨 㛩 㛪 㛫 㛬 㛭 㛮 㛯 㬒 㬡
• And then there’s Hangul like 가 간 것 게 경 과 구 규 까 낸 느 니 당 더 도 되 된 됩 들 때 라 람 렸 록 류 만 말 면 모 미 배 본 분 사 생 선 성 세 수 슷 신 십 쓰 씩 아 약 어 여 오 외 울 은 음 의 인 읽 있 잘 적 정 주 지 최 친 특 하 한 항 호 후

Feature Support Summary

Here’s what each language natively supports in its standard distribution. A white check mark isn’t quite all there, and a superscripted plus says it goes beyond what’s stated. 🌟

Unicode Properties in Regexes

Conservation of Pain & Suffering?

• There is no one “right” answer in any of this.
• Every language has its own set of Unicode ups and downs.
• I’m not suggesting anyone exchange the devil they know for one they don’t.
• That doesn’t mean all platforms are equal here. Some really are more painful to use.
• Certain common trends do emerge when you look across the language spectrum.

Trends Across Languages

• Languages with longer legacies have sizable pods of hysterical porpoises to appease, which newer languages may not.
• Compatibility with 8‐ and especially 16-bit character sets is a major drag.
• Extending the repertoire to 21 bits doesn’t resolve broken assumptions, because Unicode is much more than just “more characters”. Check out the blue boxes here to see what I mean:

   http://unicode.org/charts/case/
   http://unicode.org/charts/case/chart_Latin.html 

• Try to get the latest releases of all these languages; it makes a big difference.
• For example, this talk originally included a bunch of pain about Unicode in Python v2.7, but I deleted all the v2.7 material to get past the profanity filters.

Javascript ≠ Java’s Crypt

• Javascript still uses (or can use: the standard allow for either UTF‐16 or UCS‐2) the broken and ancient UCS‐2 — which is not a valid Unicode encoding!
• No support for named characters like \N{WHITE SMILING FACE}; you must use magic numbers like \u263A instead. Notice that’s not enough digits.
• Because literals are specified using exactly four hex digits \uXXXX, Javascript supports only ¹⁄₁₇ᵗʰ of Unicode, the 5.88% that occupies Plane Zero.
• The ᴇᴄᴍᴀScript standard wickedly defines string values not as sequences of characters, but rather as sequences of 16-bit “code units”.
• This is super‐nasty.

The UTF‐16 née UCS‐2 Curse

• Like several other languages, Javascript suffers from The UTF‐16 Curse.
• Except that Javascript has an even worse form of it, The UCS‐2 Curse. Things like charCodeAt and fromCharCode only ever deal with 16‐bit quantities, not with real, 21‐bit Unicode code points.
• Therefore, if you want to print out something like 𝒜, U+1D49C, MATHEMATICAL SCRIPT CAPITAL A, you have to specify not one character but two “char units”: "\uD835\uDC9C". 😱

   # ERROR!!
   document.write(String.fromCharCode(0x1D49C));
  
   # needed bogosity
   document.write(String.fromCharCode(0xD835,0xDC9C)); 

Testing for UTF‐16 vs UCS‐2

• You cannot write portable code because you cannot foresee which encoding is used internally.
• However, on any given Javascript interpreter, you can test for which of the two you have this way:

      var div = document.createElement("DIV"); 
      div.innerHTML = "&#0x10000;"; 
      var isUtf16 = div.firstChild.nodeValue.charCodeAt(0) == 0xd800; 

Javascript Regexes 👎

• Regex character classes are still ASCII only. You can never make \w, \d, &c work on trans‐ASCII Unicode as required by Unicode’s UTS #18 Regular Expressions in its Annex C: Compatibility Properties. This is a severe hardship.
• You will never be able to write regexes like [𝒜-𝒵] since that gets misinterpreted as [\uD835\uDC9C-\uD835\uDCB5]. Actually, you won’t even get that far:

   new RegExp("[𝒜-𝒵]", "i") returned a SyntaxError:
     RegExp constructor: invalid regular expression 

• A Javascript plugin exists that adds support for three (yes, just 3!) Unicode properties via \p{PROP} and \P{PROP}:

   http://xregexp.com/plugins/ 

  • a General Category abbreviation like \p{Sc}
  • a Script name like \p{Latin}
  • a Block property like \p{InMathematicalAlphanumericSymbols} (except that one won’t work: it lies outside the ¹⁄₁₇ᵗʰ of Unicode Javascript can even think about)

The XRegExp Plugin (cont...)

• You still have to use obscenely ＼＼𝕕𝕠𝕦𝕓𝕝𝕖𝕕  ＼＼𝕓𝕒𝕔𝕜𝕤𝕝𝕒𝕤𝕙𝕖𝕤  like

      XRegExp("[^\\p{Latin}\\p{Common}]"); 

• The XRegExp plugin doesn’t even support basic properties like Alphabetic, Uppercase, and Lowercase, all required by UTS#18 for Level 1 Unicode Support.
• It has no support for grapheme clusters whatsoever.
• No matter what, you’re still stuck with only ¹⁄₁₇ᵗʰ of Unicode — which is simply not acceptable.
• How in the world are you ever going to fully deal with a real corpus like the one I showed at the start?
• You aren’t.

PHP

• PHP⁵ is a little better than Javascript, but it will still make you pull your hair out.
• The fabled PHP⁶ “with (full) Unicode support” seems to have been cancelled.
• PHP’s “new” preg_* functions indeed use PCRE.
• However...

PHP: PCRE Issues

• Your version of PHP has to be linked with a version of PCRE that was built with Unicode support in it; this is not the default.
• You still have to always, always remember to use /u on your patterns to get the Unicode character sense instead of the “byte” sense.
• You will never(?) get \w, \d, &c to work the way UTS#18 says they must.
• Property support is limited to just two only: the General Category abbreviations and the Script properties.
• Its \X is the old broken (?>\PM\pM*), which is 🛂 not 🛂 a Unicode grapheme cluster as currently defined in UTS#18 & UTS#29.

PHP: The Ugly News

• PHP doesn’t really have native support for Unicode.
• You must always ignore normal functions like strlen, strstr, strpos, and substr
• Instead you have to remember to use mb_strlen, mb_strstr, mb_strpos, and mb_substr instead.

PHP Titlecasing

• Unlike Javascript, PHP actually makes a stab at titlecasing, something required by Unicode.
• However it does simple casemapping only, never the default Unicode casemapping, which is full casemapping. This is especially annoying with titlecasing.
• That means you can correctly titlecase dzur "\x{1F3}ur" to Dzur "\x{1F2}ur":

   # PASS
   mb_convert_case("dzur", MB_CASE_TITLE, "UTF-8"); 

• But you cannot get ß "\x{DF}" to properly become Ss, file "\x{FB01}le" to become Fi, nor ſtop "\x{FB05}op" to become Stop.

   # FAIL
   mb_convert_case("ſtop", MB_CASE_TITLE, "UTF-8"); 

Go Figure!

• Of all the languages I played around with for this talk, Go was hands down the easiest — and possibly the most pleasurable — to use UTF‐8 with.
• Go figger. 😉
• Its ᚱᚢᛋᚾᚲ heritage from Plan9 shows.
• This is hardly a bad thing.

Go: The Good News

• It’s new.
• No special syntax, declarations, functions, &c are needed to get any and all of Go to work with Unicode in its
  • source code
  • program identifiers
  • character or string literals
  • casing
  • regexes
  • I/O streams
• This is both exciting and encouraging.

Go: The Bad News

• It’s new. (Yes, I know I said that already.)
• Standard libraries for key Unicode components lag.
  • Normalization — standard library almost there.
  • Collation — is in the make, but collation is harder yet essential, because comparison underlies nearly everything we do with text.
  • Regexes — a new standard library, equivalent to Russ Cox’s RE2 library, is currently imported as exp/regexp. It provides for (simple) Unicode casefolding and includes access to the Unicode General Category and Script properties.

Go: The Ugly News

• This is Go: it doesn’t do ugly.

Ruby: The Bad News

• Fresh out of the box, Ruby at first appears so broken vis‐à‐vis Unicode as to be unusable.
• It treats your source code as being in some nonportable local encoding.
• Its String functions like upcase or capitalize won’t even look at anything but ASCII.
• Its regexes ignore your Unicode.
• It’s completely missing a whole lot of critical Unicode functionality:
  • casemapping & -folding
  • grapheme support
  • normalization
  • collation
  • text segmentation, &c &c &c.

Ruby: The Ugly News

• Every Ruby string carries around its encoding, instead of sanely unifying into Unicode internally like nearly everything else does.
• Supposedly this is to avoid round‐trip losses in certain legacy Asian encodings that would otherwise lose information due to “Han Unification”. (If you don’t know what that means, consider yourself lucky. Very lucky.)
• Because of this ugliness, you might think things like \p{Lu} are fine in your regexes because they work on your system — under a UTF‐8 locale. Now. But move the script to something laboring under CP1252 and suddenly you have an error!
• Indeed the same pattern that runs on one string may blow up on another all in the same program.
• You seem(?) to be able to avoid this by always using the /u regex flag.

Ruby: The Good News

• However, it turns out that Ruby is one of the better choices for Unicode work.
• If you want to work with Unicode in Ruby, then you must use v1.9 or better. Don’t even dream of using anything less.
• Declare your source code to be UTF‐8 with a significant comment as the first line:

    #!/usr/bin/env ruby
    #coding: utf-8 

• Ruby even allows proper Unicode identifiers.

   #!/usr/bin/env ruby
   #coding: utf-8
   𝔘𝔫𝔦𝔠𝔬𝔡𝔢   = "super";       puts 𝔘𝔫𝔦𝔠𝔬𝔡𝔢
   niño    = "boy";           puts niño
   both    = 𝔘𝔫𝔦𝔠𝔬𝔡𝔢 + niño;  puts both  
   
   # Yep, it’s superboy! 

Ruby: The Best News

• The solution to the Ruby “bad news” in the last bullet three slides ago is to install the unicode_utils gem, available via gem ininstall unicode_utils or from

   http://unicode-utils.rubyforge.org/ 

• The library includes support for Unicode normalization, character names, graphemes, combining classes, casefolding, and casemapping.
• For example, Ruby now does full and correct casemapping — provided you use it this way:

    #!/usr/bin/env ruby
    #coding: utf-8
    require 'unicode_utils'
    puts "lower is " + UnicodeUtils.downcase(str)
    puts "title is " + UnicodeUtils.titlecase(str)
    puts "upper is " + UnicodeUtils.upcase(str) 

Ruby Regexes: Good News...

• Unlike anything we’ve looked at so far, Ruby actually implements full Unicode casefolding in its regexes:

    % ruby -le 'print " WEISS" =~ /weiß/i'
    1 

• Or, in a longer example:

      #!/usr/bin/env ruby
      #coding: utf-8
  
      str = " ᾲ στο διάολο";
      puts str =~ /ᾺΙ ΣΤΟ ΔΙΆΟΛΟ/i ? "PASS" : "FAIL";
   PASS 

Ruby Regexes: ...and Bad News

• Unfortunately, regexes are just where Ruby’s Unicode support is weakest.
• Only two properties are supported: General Category and Script.
• It fails to correctly implement UTS#18’s RL1.2a: that means your \w, \d, &c won’t work right, and even \p{alphabetic} is broken.
• \w won’t work except on ASCII, even with the /u flag.
• Ruby seems to make up its own rules about what some of its properties mean instead of using Unicode’s rules. You aren’t allowed to use the same property name with a different meaning than the official one. This is a violation. 🛂

Python: (Some) Good News

• Python v3 is the first language so far discussed with any builtin support for Unicode named characters, e.g., \N{GREEK SMALL LETTER FINAL SIGMA} for U+03C2, ς.
• This is good because it frees you from magic numbers without making you use Unicode literals that might be lookalikey, e.g., \N{GREEK SMALL LETTER STIGMA} for U+03DB, ϛ.    
• However, it does not support the named sequences described by Unicode v5, like \N{LATIN CAPITAL LETTER A WITH MACRON AND GRAVE}.
• Admittedly, this is a comparitively new feature that probably hasn’t yet seen a lot of demand. Perl does support them, however:

   % perl5.14.0 -Mcharnames=:full -le 'print("\N{LATIN CAPITAL LETTER A WITH MACRON AND GRAVE}")'
   Ā̀
  
   % perl5.14.0 -Mcharnames=:full -le 'print("\N{LATIN CAPITAL LETTER A WITH MACRON AND GRAVE}")' | uniquote -v
   \N{LATIN CAPITAL LETTER A WITH MACRON}\N{COMBINING GRAVE ACCENT} 

Python: Not Good News

• Unicode in Python v2.7 is way Murphian; use v3.2 and your Unicode life will be... well, less troubled.
• Python has no support for graphemes clusters, without which it is virtually impossible to correctly process Unicode text.
• Python does not include a standard module for Unicode collation.
• Python’s standard re library is really quite deficient from a Unicode perspective.
• However, if you use re.U, then your \w, \d, &c actually do work according to UTS#18. Mostly.

Python Regexes: The Bad News

• The Python re library doesn’t offer any Unicode properties.
• It doesn’t understand graphemes.
• It doesn’t do casefolding. At all! Not simple, and not full.
• Rather, it mistakenly applies casemapping instead of casefolding, and therefore sometimes gets the wrong answer, failing to match such things as:
  • The U+00BC µ micro sign and either Greek mu, whether μ or Μ.
  • U+0175, LATIN SMALL LETTER LONG S ſ (as in the old‐style poſt) and either of the regular versions, whether S or s.
  • The regular Greek lowercase sigma, σ, whether the final sigma, ς, or the uppercase one, Σ.
  • The Greek “iota subscript”, COMBINING GREEK YPOGEGRAMMENI, ◌ͅ , and any other iota like GREEK CAPITAL LETTER IOTA Ι, GREEK SMALL LETTER IOTA ι, or GREEK PROSGEGRAMMENI ι.

Python Regexes: The Good News

• Matthew Barnett has a very interesting replacement library for the lame re module Python comes with, for both v2 and v3.
• It not only handles Unicode better (read: at all), it has lots of other super‐neat features found in no other regex library anywhere.
• It supports Unicode graphemes, boundaries, does do (simple) casefolding, doesn’t require a Unicode flag for Unicode charclasses, &c.
• Property support goes far beyond RL1.2’s required 11, with 67 properties supported as of this writing. This makes a huge difference in real processing.

    http://pypi.python.org/pypi/regex 

Python: The Ugly Bits

• Python uses the antiquated and broken UCS‐2 — er, except when it uses UCS‐4. This is a super‐massive problem. 💀
• Because you cannot count on having a wide build so you can have UCS‐4, you cannot reliably use Python on general Unicode such as the corpus I showed earlier.
• Python plays naughty guessing games with encodings, causing programs to act differently when redirected.
• Although you can have a variable named écran or el_niño, or even 東京 or 文字化け, the UCS‐2 Curse prevents having one named 𝔘𝔫𝔦𝔠𝔬𝔡𝔢, let alone 𐐔𐐯𐑅𐐨𐑉𐐯𐐻 or 𐌰𐍄𐍄𐌰‿𐌿𐌽𐍃𐌰𐍂‿𐌸𐌿‿𐌹𐌽‿𐌷𐌹𐌼𐌹𐌽𐌰𐌼. (This has just been fixed [2011-08-11])
• Atta_unsar_ þu_in_himinam is ok, though.

Testing for UCS‐2 vs UCS‐4

• As with Javascript, you cannot rely on all Python interpreters using one or another internal representation.
• However, here again you can determine which you have on any given system.

   import sys
   wide_enough = (sys.maxunicode >= 0x10FFFF)
   print("my largest character is",  sys.maxunicode) 

• On a wide build, sys.maxunicode is 0x10FFFF, whereas it is otherwise only 0xFFFF.
• That way you can abort unless you’re on a wide build.

   if not wide_enough:
       raise Exception("Narrow build lacks full Unicode support") 

Fixing Python Redirection Troubles

• You can set the PYTHONIOENCODING environment variable so it stops guessing. (This is like setting PERL_UNICODE to S.)

   $ export PYTHONIOENCODING=utf8 

• By default, stdin and stdout encoding errors raise exceptions, but those on stderr do not.
• To override all reliance on guessing or your environment, your program can do something like this (in Python v3):

   import io
   import sys
   for s in ("stdin","stdout","stderr"):
       setattr(sys, s, io.TextIOWrapper(getattr(sys, s).detach(), encoding="utf8")) 

• I can’t actually syntax‐check that previous block by doing a {!}python from within vi on the pod source, darn it!!

Python: A Complete Solution?

Even 🐍ᵛ³ + regex cannot fix all 🐍ᵛ² Unicode problems, because 🐍’s character model is inherently broken. It completely violates UTS#18’s Level 1 requirement for regular expressions — the minimal level for useful Unicode support — that the engine support Unicode characters as basic logical units independent of serialization like UTF‑*:

 % python3.2
 Python 3.2 (r32:88445, Jul 21 2011, 14:44:19)
 [GCC 4.2.1 (Apple Inc. build 5664)] on darwin
 Type "help", "copyright", "credits" or "license" for more information.
 >>> import re
 >>> g = "\N{GREEK SMALL LETTER ALPHA WITH VARIA AND YPOGEGRAMMENI}"
 >>> print(g)
ᾲ
 >>> print(re.search(r'\w', g))
 <_sre.SRE_Match object at 0x10051f988>
 >>> p = "\N{MATHEMATICAL SCRIPT CAPITAL P}"
 >>> print(p)
𝒫
 >>> print(re.search(r'\w', p))
None
 >>> print(re.search(r'..', p))   # ← 𝙏𝙃𝙄𝙎 𝙄𝙎 𝙏𝙃𝙀 𝙑𝙄𝙊𝙇𝘼𝙏𝙄𝙊𝙉 𝙍𝙄𝙂𝙃𝙏 𝙃𝙀𝙍𝙀 
<_sre.SRE_Match object at 0x10051f988>
 >>> print(len(chr(0x1D4AB)))
2 

Java

• Java’s Unicode support has a lot to be said about it, most of which wouldn’t pass muster with the potty-mouth police. 🔞
• It’s still a whole lot better than Python, though, because Java demonstrably cares about conformance with The Unicode Standard.
• Then again, Oracle pays people to work on the Java core, despite the OpenJDK. Few other languages (apart perhaps from Google’s Go) enjoy such support.

Java: The Ugly News

• Although it has several, Java’s biggest Unicode problem is that you cannot store a character in a char, nor in a Character for that matter. 🚽
• You have to use an int or a String: 👉 “int is the new char.”
• Functions that deal with chars do not deal with Unicode characters: String.length must be replaced by codePointCount, charAt by codePointAt, &c.
• Many classes simply have no “code point” (i.e., Unicode character) interfaces at all.
• This is all extremely error‐prone and unpleasant — at best.

Java: The Bad and Half‐bad

• No support for graphemes or other standard Unicode algorithms like collation and linebreaking.
• No support for named characters like \N{PILE OF POO}, so you have to put all these magic numbers in your code. But which magic numbers? It’s a 💩 situation.
• Encoding errors are silently ignored unless you work really hard.
• On the other hand, Java does support full casemapping, even though it supports only simple casefolding.

Java Tips

• Avoid the Java preprocessor! I’m talking about things like \uXXXX.
• Java has no way of indicating the encoding of a source unit interior to that file itself. You have to put it on the command line:

   % javac -encoding UTF-8 SomeanNoyinGanDillegiBlenAme.java 

• The “default” encoding if unspecified varies from one platform to the next, so if you do not specify it exclicitly, even on std{in,out,err}, your code is not portable.
• No exceptions, either, so it’s not just “write‐once & run‐nowhere”. Instead, ?it?s write?once & run?nowhere?. Have a nice day.
• Sometimes you can put a band‐aid around that problem at runtime this way:

   % java -Dfile.encoding=UTF-8 SomeanNoyinGanDillegiBlenAme 

More Java Tips

• The constructors for bridging bytes to characters all come in four forms.

   InputStreamReader(InputStream in)
   InputStreamReader(InputStream in, Charset cs)
   InputStreamReader(InputStream in, CharsetDecoder dec)
   InputStreamReader(InputStream in, String charsetName)
  
   OutputStreamWriter(OutputStream out)
   OutputStreamWriter(OutputStream out, Charset cs)
   OutputStreamWriter(OutputStream out, CharsetEncoder enc)
   OutputStreamWriter(OutputStream out, String charsetName) 

• Only one constructor out of each set of those four is usable, because the others hide errors from you.
• Do you know which is the only safe & sane one?

Java Safe Streaming

Here’s an open3 type procedure for Java, with correct constructors:

    Process
    slave_process = Runtime.getRuntime().exec("perl -CS script args");

 OutputStream
 __bytes_into_his_stdin  = slave_process.getOutputStream();

 OutputStreamWriter
   chars_into_his_stdin  = new OutputStreamWriter(
                             __bytes_into_his_stdin,
         /* DO NOT OMIT! */  Charset.forName("UTF-8").newEncoder()
                         );

 InputStream
 __bytes_from_his_stdout = slave_process.getInputStream();

 InputStreamReader
   chars_from_his_stdout = new InputStreamReader(
                             __bytes_from_his_stdout,
         /* DO NOT OMIT! */  Charset.forName("UTF-8").newDecoder()
                         );

 InputStream
 __bytes_from_his_stderr = slave_process.getErrorStream();

 InputStreamReader
   chars_from_his_stderr = new InputStreamReader(
                             __bytes_from_his_stderr,
         /* DO NOT OMIT! */  Charset.forName("UTF-8").newDecoder()
                         ); 

Java Regexes: (Finally) Some Good News

• Prior to JDK7, I would not recommend attempting to use the Java Pattern class for regexes, and would embellish any uses of Character and String with strong provisos.
• However, JDK7 goes a seriously long way toward changing that. 😇
• By bringing Java into compliance with all of UTS#18’s Level 1 requirements for the very most basic of Unicode regex handling, you can finally do basic Unicode regex handling per UTS#18.
• Java v1.7 is the first language so far discussed for which this is true.

Java Regexes: Still a Ways to Go

• However, I’d still JNI into ICU’s regex library to do serious Unicode regex work — if I had to do it in Java.
• That’s because you truly do need RL2.2’s grapheme support and RL2.7’s full property support for real Unicode work.
• That’s why this is what Google does on Android.
• The Android JNI wrappers are open source. Look for the file NativeRegEx.cpp in the public Android repository.

The JDK7 Regex Revolution

• As of v1.7, Java supports Unicode v6.0.0, with many classes fixed.
• The new \x{XXXXXX} regex escape gets around The UTF‐16 Curse. The previously broken [𝒜‑𝒵] can now be written [\x{1D49C}‑\x{1D4B5}].
• The new UNICODE_CHARACTER_CLASSES compilation flag changes the charclass abbreviations like \w &c, and also the POSIX ones like \p{alpha}, to make them do what UTS#18 says they’re supposed to do.
• There’s a new "(?U)" flag embeddable in the pattern to do the same thing.

JDK7: And That’s Not All

• Casing tests and conversions now work on all applicable code points.
  • Oops, spoke too soon: Because String.equalsIgnoreCase() still erroneously treats strings as UCS‐2 sequences not as UTF‐16, it fails to notice that 𐐔𐐇𐐝𐐀𐐡𐐇𐐓 and 𐐼𐐯𐑅𐐨𐑉𐐯𐐻 are the same. Use ICU’s CaseInsenitiveString.equals() instead for correct behavior.
• All missing properties from UTS18’s RL1.2 eleven mandatory ones have been added, including especially \p{script=SCRIPTNAME}, also usable as \p{isSCRIPTNAME} or \p{SCRIPTNAME}.
• Before they worked only on letters alone, not on Other_Alphabetic.

Perl Unicode Gotchas

• I’ve already given something like four hours of talks this week on Unicode in Perl. Since we’re already over our time limit, I’ll mostly refer you to those materials.
• You may wish to set your PERL_UNICODE envariable to "SA", or in certain very limited & one‐shot cases, even to "SAD".
• The short story is that you will spare yourself a great deal of pain and suffering if you start every Perl program this way:

   use v5.14;   # many, 𝘮𝘢𝘯𝘺 Unicode fixes
   use utf8;    # so source code is UTF-8
   use strict;
   use warnings;
  
   use autodie; ## 𝘽𝙐𝙂! conflicts with 𝘶𝘴𝘦 𝘰𝘱𝘦𝘯 below!!
  
   use charnames qw< :full >;     # standard \N{charnames}
  
   use open qw< :std :encoding(UTF-8) >;  # default streams 

Perl Unicode Specialties

• Perl meets RL2.7 by supporting all standard Unicode properties, including all name and value aliases. But Perl goes further: it even lets you create your very own customized properties.
• Same story with \N{NAMED CHARACTER}: you can create your own customized versions of those, too. I use this all the time.
• Not only does Perl not restrict your code points to 16 bits, it doesn’t even restrict them to Unicode’s own 21 bits. If you can’t think of something to do with 72 bits of characters, I know people who can.
• Perl’s standard module implementing the Unicode Collation Algorithm — part of every distribution since v5.7 and now as of v5.14 with locale support, too — makes it especially easy to create customized sorts, run accent‐insensitive string comparisons and searches, and much more. I use this nearly every day.

🏁 And Our Winner Is...

• Look back at the two charts of feature summaries on slides 7 & 8. We knew where we were going right from the start: 💪 Perl still wins this round.
• Even though 🐪 offers the most complete Unicode support of all these languages, that doesn’t mean it’s without its bumps: there’s still plenty of work to be done.
• No matter what language you use, everyone needs to learn basic things about Unicode, and perhaps unlearn some basic things, too: myths such as Unicode is “like ASCII but with more characters” or that it’s “just for non‐English”.
• If 🐪 isn’t your favorite language, this is a great opportunity for ambitious and enterprising 🐣 programmers to bring 🐍 💎 ☕ up to speed at ((more) fully) supporting The Unicode Standard.
• Perl avoided certain Unicode mistakes other languages made (like Java, for one) by learning from others’ mistakes; maybe you can see things Perl has done wrong with Unicode and similarly improve your own language. 👓

Good Luck 🍀 and Good Night 🌃

• Even if you don’t use Perl, the score or so of Unicode tools I wrote that are in the Unicode::Tussle bundle from CPAN will really make your Unicode life a lot easier.

   http://search.cpan.org/perldoc?Unicode::Tussle
  
   % sudo perl -MCPAN -e 'install Unicode::Tussle' 

• Brief demos of these tools are included in my other conference materials, plus many more such are included with the module documentation.
• Thanks very much to Brian Foy for putting that CPAN bundle together for me, and for getting me to give this talk in the first place. 👏

  𝔢𝔵𝔢𝔲𝔫𝔱 𝔬𝔪𝔫𝔢𝔰

  🍸

Appendix 1: Font suggestions

• I recommend two free fonts from George Douros at users.teilar.gr/~g1951d/ known to work with this presentation: his Alfios font for regular text, and his Symbola font for fancy emoji. If any of these don’t look right to you, you probably need to supplement your system fonts:
  • Ligatures: fi ffi ff ffl fl β ẞ ſt st
  • Math letters: 𝒜 𝒟 𝔅 𝔎 𝔼 𝔽
  • Gothic & Deseret: 𐌸𐌼𐌽𐍂, 𐐔𐐯𐑅𐐨𐑉𐐯𐐻
  • Symbols: ✔ ✅ 🐪 📖 🛂 🐍
  • Emoticons: 😇 😈 😉 😨 😭 😱
  • Upside‐down: ¡pɐəɥ ɹnoʎ uo ƃuᴉpuɐʇs ʎq sᴉɥʇ pɐəᴚ
  • Combining characters: ◌̂,◌̃,◌⃞,◌̲,◌︀,◌̵,◌̷
• The last line with combining characters is especially hard to get to look right. You may find that the shareware font Everson Mono works when all else fails.

Appendix 2: Tools

• I wrote a huge bucketful of tools to make your life with Unicode not just easier, but more fun.
• These are available in the Unicode::Tussle bundle from CPAN, where they come with documentation, or you can get them individually from training.perl.com/scripts if you’d prefer.
• Thanks very much to brian d foy for putting that CPAN bundle together for me.

Contact Information

• I’m Tom Christiansen. You can reach me at tchrist@perl.com.
• All three talks, as well as these instructions, are available from training.perl.com/OSCON2011.