Re: UTF-8 from Day One

--===============26540253627665233==
Content-Type: multipart/signed; micalg=pgp-sha1;
	protocol="application/pgp-signature";
	boundary="=-XoEOyLSHGQAstcNCaCvx"


--=-XoEOyLSHGQAstcNCaCvx
Content-Type: text/plain
Content-Transfer-Encoding: quoted-printable

I would agree with shooting for full UTF support. I think what we need
to accomplish this is a good howto written. If we handle the base
component changes, and get it to the point where adding UTF support in
ebuilds is trivial, then it shouldn't be much of a task at all.

Thanks,
Graham


On Sun, 2003-08-31 at 10:13, Low Zhen Lin wrote:
> Why UTF-8 support should be in Zynot from day one.
> -- Low Zhen Lin (zhenlin on #zynot-dev)
> =20
>=20
> 0. Preface
> =20
> Firstly, you might be wondering, what is UTF-8? Well, RTFM! Really!
> man 7 utf-8 will bring up the document.
> =20
> For those of you who are not fortunate enough to be able to bring up
> that man page, here is my explanation:
> =20
> UTF-8 is a variable-length encoding format for the Universal Character
> Set. The Universal Character Set, as defined in ISO
> 10646 requires a 31-bit address space for each character. The obvious
> encoding would be a 32-bit integer - however - consider this: all
> characters will take up 4 bytes, compared to ASCII, where all
> characters are encoded in 7 bits! This would be a gross waste of space
> - if using Latin-1, 300% of your document's bytes will be 0x00!
> =20
> Enter UTF-8. It uses 1 byte for 7 bit ASCII characters (0x00 to 0x7F),
> 2 bytes for characters with code points between 0x80 and 0x07FF, 3
> bytes for characters between 0x800 and 0xFFFF. Good for most
> alphabet-based languages - Basic Latin takes up one byte; Latin-1
> Supplement, Latin Extended-A, Latin Extended-B, IPA Extensions,
> Spacing Modifier Letters, Combining Diacritical Marks, Greek,
> Cyrillic, Armenian, Hebrew, Arabic, Syriac and Thaana takes up two
> bytes.
> =20
> Of course UTF-8 is not the perfect solution for everything. For mostly
> Greek documents, it is probably unacceptable to have every character
> take up 2 bytes when it can be encoded in one. Another example is a
> mostly CJK document - most characters will take up 3 bytes, as opposed
> to UTF-16 which only takes up 2 bytes.
>=20
>=20
> 1. Reasoning=20
> So, you might be thinking, why we need UTF-8 support from day one?
> Well, look here. [http://bugs.gentoo.org/show_bug.cgi?id=3D18375] Adding
> in UTF-8 support later will be problematic. I know - I am working on
> getting as many packages as possible to work with the wide-character
> version of ncurses, ncursesw. If we do not have support for UTF-8 from
> day one, it will definitely be a headache to enable it later.
> =20
> Another thing you might be thinking, why do we even need UTF-8
> support? Well, that would be for a variety of reasons - for one thing,
> it is a pull factor, it puts us on par with Windows NT/2000/XP and Mac
> OS X, as well as Lucent/Bell Lab's next generation OS, Plan 9; for
> another, it allows us to deal with most of the world's commonly-used
> characters in one encoding - it's called Universal Character Set for a
> reason.
> =20
>=20
> 2. Proposal
> =20
> I am proposing that, before the user starts bootstrap, the user must
> decide whether or not he/she wants UTF-8 support. But not after. The
> user cannot decide "I want UTF-8 from now on" and just activate a USE
> flag - because it may require many packages to be rebuilt.
>=20
>=20
> 3. Implementation
> =20
> The following changes are involved:
> =20
>       * Distributing Unicode fonts. (Bitstream Cyberbit for instance)
>       * Defining and using xx_XX.UTF-8 versions of all locales. (Do in
>         glibc ebuild)
>       * Using ncursesw instead of ncurses. (--enable-widec)
>               * Changing configure scripts to recognise ncursesw as
>                 well as ncurses. (Not neccessary if libncurses is
>                 linked to libncursesw, but that is a _workaround_, not
>                 a _fix_)
>       * Applying RH9's patches to slang, and install as default.
>               * Patch apps that are broken by slight API change.
>       * Applying RH9's patches that are relevant to UTF-8 support to
>         appropriate packages
>       * Making sure unicode_start is run on every console.
>       * For x86 platforms:
>               * Using fbiterm for consoles.
>               * Alternatively, distribute a comprehensive font for use
>                 in the EGA/VGA console.
>        =20
> There are many other things to do, an outdated list can be found in
> the LDP Unicode HOWTO, chapter 4.=20
> [http://tldp.org/HOWTO/Unicode-HOWTO-4.html]
>=20
>=20
> 4. Problems
> =20
> Assumptions such as 'one byte per character' are broken by UTF-8, so
> porting applications will be tough. See the LDP Unicode HOWTO, chapter
> 6. [http://tldp.org/HOWTO/Unicode-HOWTO-6.html]
> =20
>=20
> 5. Acceptable Concessions
> =20
> It is acceptable to only have support for reading and writing UTF-8,
> but not displaying it.
>=20
> It is also acceptable to build applications against wide-character
> enabled libraries but not adding UTF-8 I/O support to the application.
> =20
>=20
>=20
>=20
> ______________________________________________________________________
> _______________________________________________
> zynot-dev mailing list
> [email protected]
> http://lists.zynot.org/mailman/listinfo/zynot-dev

--=-XoEOyLSHGQAstcNCaCvx
Content-Type: application/pgp-signature; name=signature.asc
Content-Description: This is a digitally signed message part

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.2 (GNU/Linux)

iD8DBQA/VhULAuxgtoh9rPYRAso+AKDrgNg/t387/Qzl2ec9gdFH4oeIYACgi+I7
jB445usQwDK1CDky75g9ZHU=
=vjiD
-----END PGP SIGNATURE-----

--=-XoEOyLSHGQAstcNCaCvx--


--===============26540253627665233==
Content-Type: text/plain; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Disposition: inline

_______________________________________________
zynot-dev mailing list
[email protected]
http://lists.zynot.org/mailman/listinfo/zynot-dev

--===============26540253627665233==--