UTF-8 from Day One

This is a multi-part message in MIME format.
--===============36505806603213542==
Content-Type: multipart/alternative;
	boundary="------------010008040701060308090006"

This is a multi-part message in MIME format.
--------------010008040701060308090006
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit


    Why UTF-8 support should be in Zynot from day one.

-- Low Zhen Lin (zhenlin on #zynot-dev)
 


      0. Preface

 
Firstly, you might be wondering, what is UTF-8? Well, RTFM! Really! man 
7 utf-8 will bring up the document.
 
For those of you who are not fortunate enough to be able to bring up 
that man page, here is my explanation:
 
UTF-8 is a variable-length encoding format for the Universal Character 
Set. The Universal Character Set, as defined in ISO
10646 requires a 31-bit address space for each character. The obvious 
encoding would be a 32-bit integer - however - consider this: all 
characters will take up 4 bytes, compared to ASCII, where all characters 
are encoded in 7 bits! This would be a gross waste of space - if using 
Latin-1, 300% of your document's bytes will be 0x00!
 
Enter UTF-8. It uses 1 byte for 7 bit ASCII characters (0x00 to 0x7F), 2 
bytes for characters with code points between 0x80 and 0x07FF, 3 bytes 
for characters between 0x800 and 0xFFFF. Good for most alphabet-based 
languages - Basic Latin takes up one byte; Latin-1 Supplement, Latin 
Extended-A, Latin Extended-B, IPA Extensions, Spacing Modifier Letters, 
Combining Diacritical Marks, Greek, Cyrillic, Armenian, Hebrew, Arabic, 
Syriac and Thaana takes up two bytes.
 
Of course UTF-8 is not the perfect solution for everything. For mostly 
Greek documents, it is probably unacceptable to have every character 
take up 2 bytes when it can be encoded in one. Another example is a 
mostly CJK document - most characters will take up 3 bytes, as opposed 
to UTF-16 which only takes up 2 bytes.


      1. Reasoning 


So, you might be thinking, why we need UTF-8 support from day one? Well, 
look here. [http://bugs.gentoo.org/show_bug.cgi?id=18375] Adding in 
UTF-8 support later will be problematic. I know - I am working on 
getting as many packages as possible to work with the wide-character 
version of ncurses, ncursesw. If we do not have support for UTF-8 from 
day one, it will definitely be a headache to enable it later.
 
Another thing you might be thinking, why do we even need UTF-8 support? 
Well, that would be for a variety of reasons - for one thing, it is a 
pull factor, it puts us on par with Windows NT/2000/XP and Mac OS X, as 
well as Lucent/Bell Lab's next generation OS, Plan 9; for another, it 
allows us to deal with most of the world's commonly-used characters in 
one encoding - it's called Universal Character Set for a reason.
 


      2. Proposal

 
I am proposing that, before the user starts bootstrap, the user must 
decide whether or not he/she wants UTF-8 support. But not after. The 
user cannot decide "I want UTF-8 from now on" and just activate a USE 
flag - because it may require many packages to be rebuilt.


      3. Implementation

 
The following changes are involved:
 

    * Distributing Unicode fonts. (Bitstream Cyberbit for instance)
    * Defining and using xx_XX.UTF-8 versions of all locales. (Do in
      glibc ebuild)
    * Using ncursesw instead of ncurses. (--enable-widec)
          o Changing configure scripts to recognise ncursesw as well as
            ncurses. (Not neccessary if libncurses is linked to
            libncursesw, but that is a _workaround_, not a _fix_)
    * Applying RH9's patches to slang, and install as default.
          o Patch apps that are broken by slight API change.
    * Applying RH9's patches that are relevant to UTF-8 support to
      appropriate packages
    * Making sure unicode_start is run on every console.
    * For x86 platforms:
          o Using fbiterm for consoles.
          o Alternatively, distribute a comprehensive font for use in
            the EGA/VGA console.

There are many other things to do, an outdated list can be found in the 
LDP Unicode HOWTO, chapter 4.  [http://tldp.org/HOWTO/Unicode-HOWTO-4.html]


      4. Problems

 
Assumptions such as 'one byte per character' are broken by UTF-8, so 
porting applications will be tough. See the LDP Unicode HOWTO, chapter 
6. [http://tldp.org/HOWTO/Unicode-HOWTO-6.html]
 


      5. Acceptable Concessions

 
It is acceptable to only have support for reading and writing UTF-8, but 
not displaying it.

It is also acceptable to build applications against wide-character 
enabled libraries but not adding UTF-8 I/O support to the application.
 


--------------010008040701060308090006
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
<html>
<head>
  <meta http-equiv=3D"Content-Type" content=3D"text/html;charset=3DUTF-8"=
>
  <title></title>
</head>
<body>
<meta http-equiv=3D"Content-Type" content=3D"text/html;charset=3DUTF-8">
<title></title>
<h2>Why UTF-8 support should be in Zynot from day one.</h2>
-- Low Zhen Lin (zhenlin on #zynot-dev)<br>
=C2=A0<br>
<h3>0. Preface</h3>
=C2=A0<br>
Firstly, you might be wondering, what is UTF-8? Well, RTFM! Really! <span
 style=3D"font-family: monospace;">man 7 utf-8</span> will bring up the
document.<br>
=C2=A0<br>
For those of you who are not fortunate enough to be able to bring up
that man page, here is my explanation:<br>
=C2=A0<br>
UTF-8 is a variable-length encoding format for the Universal Character
Set. The Universal Character Set, as defined in ISO<br>
10646 requires a 31-bit address space for each character. The obvious
encoding would be a 32-bit integer - however - consider this: all
characters will take up 4 bytes, compared to ASCII, where all
characters are encoded in 7 bits! This would be a gross waste of space
- if using Latin-1, 300% of your document's bytes will be 0x00!<br>
=C2=A0<br>
Enter UTF-8. It uses 1 byte for 7 bit ASCII characters (0x00 to 0x7F),
2 bytes for characters with code points between 0x80 and 0x07FF, 3
bytes for characters between 0x800 and 0xFFFF. Good for most
alphabet-based languages - Basic Latin takes up one byte; Latin-1
Supplement, Latin Extended-A, Latin Extended-B, IPA Extensions, Spacing
Modifier Letters, Combining Diacritical Marks, Greek, Cyrillic,
Armenian, Hebrew, Arabic, Syriac and Thaana takes up two bytes.<br>
=C2=A0<br>
Of course UTF-8 is not the perfect solution for everything. For mostly
Greek documents, it is probably unacceptable to have every character
take up 2 bytes when it can be encoded in one. Another example is a
mostly CJK document - most characters will take up 3 bytes, as opposed
to UTF-16 which only takes up 2 bytes.<br>
<span style=3D"font-weight: bold;"><br>
</span>
<h3>1. Reasoning=C2=A0</h3>
<br>
So, you might be thinking, why we need UTF-8 support from day one?
Well, look here. [<a href=3D"http://bugs.gentoo.org/show_bug.cgi?id=3D183=
75">http://bugs.gentoo.org/show_bug.cgi?id=3D18375</a>]
Adding in UTF-8 support later will be problematic. I know - I am
working on getting as many packages as possible to work with the
wide-character version of ncurses, ncursesw. If we do not have support
for UTF-8 from day one, it will definitely be a headache to enable it
later.<br>
=C2=A0<br>
Another thing you might be thinking, why do we even need UTF-8 support?
Well, that would be for a variety of reasons - for one thing, it is a
pull factor, it puts us on par with Windows NT/2000/XP and Mac OS X, as
well as Lucent/Bell Lab's next generation OS, Plan 9; for another, it
allows us to deal with most of the world's commonly-used characters in
one encoding - it's called Universal Character Set for a reason.<br>
=C2=A0<br>
<h3>2. Proposal</h3>
=C2=A0<br>
I am proposing that, before the user starts bootstrap, the user must
decide whether or not he/she wants UTF-8 support. But not after. The
user cannot decide "I want UTF-8 from now on" and just activate a USE
flag - because it may require many packages to be rebuilt.<br>
<br>
<h3>3. Implementation</h3>
=C2=A0<br>
The following changes are involved:<br>
=C2=A0<br>
<ul>
  <li>Distributing Unicode fonts. (Bitstream Cyberbit for instance)</li>
  <li>Defining and using xx_XX.UTF-8 versions of all locales. (Do in
glibc ebuild)</li>
  <li>Using ncursesw instead of ncurses. (--enable-widec)</li>
  <ul>
    <li>Changing configure scripts to recognise ncursesw as well as
ncurses. (Not neccessary if libncurses is linked to libncursesw, but
that is a _workaround_, not a _fix_)</li>
  </ul>
  <li>Applying RH9's patches to slang, and install as default.</li>
  <ul>
    <li>Patch apps that are broken by slight API change.</li>
  </ul>
  <li>Applying RH9's patches that are relevant to UTF-8 support to
appropriate packages<br>
  </li>
  <li>Making sure unicode_start is run on every console.</li>
  <li>For x86 platforms:</li>
  <ul>
    <li>Using fbiterm for consoles.</li>
    <li>Alternatively, distribute a comprehensive font for use in the
EGA/VGA console.<br>
    </li>
  </ul>
</ul>
There are many other things to do, an outdated list can be found in the
LDP Unicode HOWTO, chapter 4.=C2=A0 [<a
 href=3D"http://tldp.org/HOWTO/Unicode-HOWTO-4.html">http://tldp.org/HOWT=
O/Unicode-HOWTO-4.html</a>]<br>
<br>
<h3>4. Problems</h3>
=C2=A0<br>
Assumptions such as 'one byte per character' are broken by UTF-8, so
porting applications will be tough. See the LDP Unicode HOWTO, chapter
6. [<a href=3D"http://tldp.org/HOWTO/Unicode-HOWTO-6.html">http://tldp.or=
g/HOWTO/Unicode-HOWTO-6.html</a>]<br>
=C2=A0<br>
<h3>5. Acceptable Concessions</h3>
=C2=A0<br>
It is acceptable to only have support for reading and writing UTF-8,
but not displaying it.<br>
<br>
It is also acceptable to build applications against wide-character
enabled libraries but not adding UTF-8 I/O support to the application.<br=
>
=C2=A0<br>
<br>
</body>
</html>

--------------010008040701060308090006--


--===============36505806603213542==
Content-Type: text/plain; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Disposition: inline

_______________________________________________
zynot-dev mailing list
[email protected]
http://lists.zynot.org/mailman/listinfo/zynot-dev

--===============36505806603213542==--