Unicode support pre alpha

Hi Y'all,

For the adventurous...

We now have Basic (Level 1) Unicode Support in Qi
(See http://unicode.org/reports/tr18/ for details).

If anyone is interested to break it in, you're welcome.
It's mostly untested yet except for the posix style char class
parsers such as alnum,alpha,lower etc. I'd appreciate help in
writing test code.

You need to define:

    #define BOOST_SPIRIT_UNICODE

to enable support. Like ascii and iso8859_1, there are placeholders
in its namespace, e.g.:

     using boost::spirit::unicode::alpha;

The char type expected is a boost::uint32_t.

In general, you don't deal with that directly and instead use
UTF-8, so you use the iterators provided by John Maddock (from
Boost regex), to convert from UTF-8 to boost::uint32. The header:

     #include <boost/regex/pending/unicode_iterator.hpp>

exposes these. I should probably make that a first class spirit
citizen. Anyway, here's what you need:

     template <class BaseIterator, class U32Type = ::boost::uint32_t>
     class u8_to_u32_iterator;

     "Adapts sequence of UTF-8 code points to "look like" a
      sequence of UTF-32."

Support for no-case handling is tricky because some code-points do
not have a 1-1 correspondence (i.e. may shrink or expand the string).
For now, no_case is supported for code-points that map to single
code-points only.

In addition to the usual posix style alpha,digit,alnum etc. we have
these new placeholders:

///////////////////////////////////////////////////////////////////////////
//  Unicode Major Categories
///////////////////////////////////////////////////////////////////////////
     letter
     mark
     number
     separator
     other
     punctuation
     symbol

///////////////////////////////////////////////////////////////////////////
//  Unicode General Categories
///////////////////////////////////////////////////////////////////////////
     uppercase_letter
     lowercase_letter
     titlecase_letter
     modifier_letter
     other_letter

     nonspacing_mark
     enclosing_mark
     spacing_mark

     decimal_number
     letter_number
     other_number

     space_separator
     line_separator
     paragraph_separator

     control
     format
     private_use
     surrogate
     unassigned

     dash_punctuation
     open_punctuation
     close_punctuation
     connector_punctuation
     other_punctuation
     initial_punctuation
     final_punctuation

     math_symbol
     currency_symbol
     modifier_symbol
     other_symbol

///////////////////////////////////////////////////////////////////////////
//  Unicode Derived Categories
///////////////////////////////////////////////////////////////////////////
     alphabetic
     uppercase
     lowercase
     white_space
     hex_digit
     noncharacter_code_point
     default_ignorable_code_point

///////////////////////////////////////////////////////////////////////////
//  Unicode Scripts
///////////////////////////////////////////////////////////////////////////
     arabic
     imperial_aramaic
     armenian
     avestan
     balinese
     bamum
     bengali
     bopomofo
     braille
     buginese
     buhid
     canadian_aboriginal
     carian
     cham
     cherokee
     coptic
     cypriot
     cyrillic
     devanagari
     deseret
     egyptian_hieroglyphs
     ethiopic
     georgian
     glagolitic
     gothic
     greek
     gujarati
     gurmukhi
     hangul
     han
     hanunoo
     hebrew
     hiragana
     katakana_or_hiragana
     old_italic
     javanese
     kayah_li
     katakana
     kharoshthi
     khmer
     kannada
     kaithi
     tai_tham
     lao
     latin
     lepcha
     limbu
     linear_b
     lisu
     lycian
     lydian
     malayalam
     mongolian
     meetei_mayek
     myanmar
     nko
     ogham
     ol_chiki
     old_turkic
     oriya
     osmanya
     phags_pa
     inscriptional_pahlavi
     phoenician
     inscriptional_parthian
     rejang
     runic
     samaritan
     old_south_arabian
     saurashtra
     shavian
     sinhala
     sundanese
     syloti_nagri
     syriac
     tagbanwa
     tai_le
     new_tai_lue
     tamil
     tai_viet
     telugu
     tifinagh
     tagalog
     thaana
     thai
     tibetan
     ugaritic
     vai
     old_persian
     cuneiform
     yi
     inherited
     common
     unknown

Regards,
-- 
Joel de Guzman
http://www.boostpro.com
http://spirit.sf.net
http://www.facebook.com/djowel

Meet me at BoostCon
http://www.boostcon.com/home
http://www.facebook.com/boostcon




------------------------------------------------------------------------------
The Planet: dedicated and managed hosting, cloud storage, colocation
Stay online with enterprise data centers and the best network in the business
Choose flexible plans and management services without long-term contracts
Personal 24x7 support from experience hosting pros just a phone call away.
http://p.sf.net/sfu/theplanet-com