Re: Unicode support pre alpha

On Thu, Feb 4, 2010 at 6:09 PM, Joel de Guzman
<[email protected]> wrote:
> Hi Y'all,
>
> For the adventurous...
>
> We now have Basic (Level 1) Unicode Support in Qi
> (See http://unicode.org/reports/tr18/ for details).
>
> If anyone is interested to break it in, you're welcome.
> It's mostly untested yet except for the posix style char class
> parsers such as alnum,alpha,lower etc. I'd appreciate help in
> writing test code.
>
> You need to define:
>
>    #define BOOST_SPIRIT_UNICODE
>
> to enable support. Like ascii and iso8859_1, there are placeholders
> in its namespace, e.g.:
>
>     using boost::spirit::unicode::alpha;
>
> The char type expected is a boost::uint32_t.
>
> In general, you don't deal with that directly and instead use
> UTF-8, so you use the iterators provided by John Maddock (from
> Boost regex), to convert from UTF-8 to boost::uint32. The header:
>
>     #include <boost/regex/pending/unicode_iterator.hpp>
>
> exposes these. I should probably make that a first class spirit
> citizen. Anyway, here's what you need:
>
>     template <class BaseIterator, class U32Type = ::boost::uint32_t>
>     class u8_to_u32_iterator;
>
>     "Adapts sequence of UTF-8 code points to "look like" a
>      sequence of UTF-32."
>
> Support for no-case handling is tricky because some code-points do
> not have a 1-1 correspondence (i.e. may shrink or expand the string).
> For now, no_case is supported for code-points that map to single
> code-points only.
>
> In addition to the usual posix style alpha,digit,alnum etc. we have
> these new placeholders:
>
> ///////////////////////////////////////////////////////////////////////////
> //  Unicode Major Categories
> ///////////////////////////////////////////////////////////////////////////
>     letter
>     mark
>     number
>     separator
>     other
>     punctuation
>     symbol
>
> ///////////////////////////////////////////////////////////////////////////
> //  Unicode General Categories
> ///////////////////////////////////////////////////////////////////////////
>     uppercase_letter
>     lowercase_letter
>     titlecase_letter
>     modifier_letter
>     other_letter
>
>     nonspacing_mark
>     enclosing_mark
>     spacing_mark
>
>     decimal_number
>     letter_number
>     other_number
>
>     space_separator
>     line_separator
>     paragraph_separator
>
>     control
>     format
>     private_use
>     surrogate
>     unassigned
>
>     dash_punctuation
>     open_punctuation
>     close_punctuation
>     connector_punctuation
>     other_punctuation
>     initial_punctuation
>     final_punctuation
>
>     math_symbol
>     currency_symbol
>     modifier_symbol
>     other_symbol
>
> ///////////////////////////////////////////////////////////////////////////
> //  Unicode Derived Categories
> ///////////////////////////////////////////////////////////////////////////
>     alphabetic
>     uppercase
>     lowercase
>     white_space
>     hex_digit
>     noncharacter_code_point
>     default_ignorable_code_point
>
> ///////////////////////////////////////////////////////////////////////////
> //  Unicode Scripts
> ///////////////////////////////////////////////////////////////////////////
>     arabic
>     imperial_aramaic
>     armenian
>     avestan
>     balinese
>     bamum
>     bengali
>     bopomofo
>     braille
>     buginese
>     buhid
>     canadian_aboriginal
>     carian
>     cham
>     cherokee
>     coptic
>     cypriot
>     cyrillic
>     devanagari
>     deseret
>     egyptian_hieroglyphs
>     ethiopic
>     georgian
>     glagolitic
>     gothic
>     greek
>     gujarati
>     gurmukhi
>     hangul
>     han
>     hanunoo
>     hebrew
>     hiragana
>     katakana_or_hiragana
>     old_italic
>     javanese
>     kayah_li
>     katakana
>     kharoshthi
>     khmer
>     kannada
>     kaithi
>     tai_tham
>     lao
>     latin
>     lepcha
>     limbu
>     linear_b
>     lisu
>     lycian
>     lydian
>     malayalam
>     mongolian
>     meetei_mayek
>     myanmar
>     nko
>     ogham
>     ol_chiki
>     old_turkic
>     oriya
>     osmanya
>     phags_pa
>     inscriptional_pahlavi
>     phoenician
>     inscriptional_parthian
>     rejang
>     runic
>     samaritan
>     old_south_arabian
>     saurashtra
>     shavian
>     sinhala
>     sundanese
>     syloti_nagri
>     syriac
>     tagbanwa
>     tai_le
>     new_tai_lue
>     tamil
>     tai_viet
>     telugu
>     tifinagh
>     tagalog
>     thaana
>     thai
>     tibetan
>     ugaritic
>     vai
>     old_persian
>     cuneiform
>     yi
>     inherited
>     common
>     unknown

This is quite *awesome*, just what I needed to be able to continue a
few things.  Will give it a try when I get time soon.

------------------------------------------------------------------------------
The Planet: dedicated and managed hosting, cloud storage, colocation
Stay online with enterprise data centers and the best network in the business
Choose flexible plans and management services without long-term contracts
Personal 24x7 support from experience hosting pros just a phone call away.
http://p.sf.net/sfu/theplanet-com
_______________________________________________
Spirit-devel mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/spirit-devel