Re: Unicode support pre alpha

Joel de Guzman wrote:
 > Hi Y'all,
 >
 > For the adventurous...
 >
 > We now have Basic (Level 1) Unicode Support in Qi
 > (See http://unicode.org/reports/tr18/ for details).
 >
 > If anyone is interested to break it in, you're welcome.
 > It's mostly untested yet except for the posix style char class
 > parsers such as alnum,alpha,lower etc. I'd appreciate help in
 > writing test code.
 >
 > You need to define:
 >
 >     #define BOOST_SPIRIT_UNICODE
 >
 > to enable support. Like ascii and iso8859_1, there are placeholders
 > in its namespace, e.g.:
 >
 >      using boost::spirit::unicode::alpha;
 >
 > The char type expected is a boost::uint32_t.
 >
 > In general, you don't deal with that directly and instead use
 > UTF-8, so you use the iterators provided by John Maddock (from
 > Boost regex), to convert from UTF-8 to boost::uint32. The header:
 >
 >      #include <boost/regex/pending/unicode_iterator.hpp>
 >
 > exposes these. I should probably make that a first class spirit
 > citizen. Anyway, here's what you need:
 >
 >      template <class BaseIterator, class U32Type = ::boost::uint32_t>
 >      class u8_to_u32_iterator;
 >
 >      "Adapts sequence of UTF-8 code points to "look like" a
 >       sequence of UTF-32."
 >
 > Support for no-case handling is tricky because some code-points do
 > not have a 1-1 correspondence (i.e. may shrink or expand the string).
 > For now, no_case is supported for code-points that map to single
 > code-points only.
 >
 > In addition to the usual posix style alpha,digit,alnum etc. we have
 > these new placeholders:
 >
 > 
///////////////////////////////////////////////////////////////////////////
 > //  Unicode Major Categories
 > 
///////////////////////////////////////////////////////////////////////////
 >      <snip/>
 > 
///////////////////////////////////////////////////////////////////////////
 > //  Unicode General Categories
 > 
///////////////////////////////////////////////////////////////////////////
 >      <snip/>
 > 
///////////////////////////////////////////////////////////////////////////
 > //  Unicode Derived Categories
 > 
///////////////////////////////////////////////////////////////////////////
 >      <snip/>
 > 
///////////////////////////////////////////////////////////////////////////
 > //  Unicode Scripts
 > 
///////////////////////////////////////////////////////////////////////////
 >      <snip/>

I have a library for Unicode support (so-called Boost.Unicode), 
initially developed as a GSoC in 2009, that I plan to submit to Boost in 
the coming months.

My aim (not necessarily achieved by the first milestone) is to provide 
all the necessary tools for a library such as Spirit to be able to 
comply with level 1 or 2 of support.

For example at the lowest level it provides an access to the Unicode 
Character Database which provides various properties from a code point.
The database is generated from Unicode.org data parsed using Spirit 
Classic, and it is intended to support versioning and even in the future 
leverage other databases already provided by the OS or platform.

It also provides support for on-the-fly conversion between encodings, 
segmentation (code points, combining character sequences, graphemes, 
...) and normalization with iterator adaptors in a similar way to John 
Maddock's iterators.

I would like to know if you are aware of this library and if you believe 
it could be a viable candidate for Unicode support, or if Spirit prefers 
to maintain its own.

A not up to date documentation can be found here: 
<http://blogloufoque.free.fr/unicode/doc/html/>

------------------------------------------------------------------------------
Download Intel&#174; Parallel Studio Eval
Try the new software tools for yourself. Speed compiling, find bugs
proactively, and fine-tune applications for parallel performance.
See why Intel Parallel Studio got high marks during beta.
http://p.sf.net/sfu/intel-sw-dev