Re: Internationalisation

This is a multi-part message in MIME format.
--===============2222717750721890078==
Content-Type: multipart/alternative;
 boundary="------------020303080309060606010401"

This is a multi-part message in MIME format.
--------------020303080309060606010401
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit

On 06/03/14 07:00, Markus Dolze wrote:
> Hi,
>
> character encoding - where should I begin?
A big subject I agree, however I am trying to provide a solution to a 
relatively simple specific problem.
>
> It is true that LCDd does not handle anything else than single byte
> characters as input from clients.
>
> However, it does not assume any encoding - other than it contains
> characters that can be parsed into commands.
There are some implied assumptions:

 1. Commands are in ASCII so an encoding has to be a superset of ASCII
    (e.g. ISO/IEC 8859-n or UTF-8).
 2. Text to be displayed will map to the appropriate glyphs on the display.

>
> Any data is passed to the drivers 'as is'. Some drivers map that (single
> byte) characters to values in CGROM of the display. The Crystalfontz and
> HD4480 drivers e.g. assume ISO 8859-1 as input by default and can handle
> ISO 8859-5, CP-1251, KOI8-R as well.
>
> The picolcd driver does not do any character mapping.
>
>
> Currently I am fine with this, except the developer's guide does not
> talk about it. Clients are responsible for sending single byte
> characters to LCDd (whatever encoding).
The graphic drivers have the capability to display any Unicode character 
so the input needs to move away from single byte characters. UTF-8 
allows encoding any Unicode character into a byte stream.
>
> I am not agaist using UTF-8 as input. If we want to use it, my opinion
> is not to apply it to some 'widget text value', but to the whole input
> before it is parsed.
I put the conversion on the widget text so that debug etc. reports the 
input as sent. It should work elsewhere if that is preferred. Conversion 
should have no effect on commands, they are ASCII so will not be changed 
for any encoding lightly to be used.
>
> By using libiconv we may get rid of mapping non ISO 8859-1 encoded input
> to CGROM values. Or not.
Depends how well the CGROM characters map to an encoding supported by 
the system, it may be possible to generate a mapping table for libiconv 
for each display.
>
> However, auto-detecting the system locale only works if client and
> server run on the same system and share the same locale (which might not
> be true for LCDd started at boot time and a client that runs in a user
> session).
I've allowed the locale to be set from the configuration, maybe the 
comment needs more explanation.
>
> Library dependencies and compatibility are another topic for some future
> discussion.
>
> Regards,
> Markus
>
>
> On 27.02.2014 21:47, Martin Tharby Jones wrote:
>> Hello,
>>
>> I've attached a patch which provides a first step towards
>> internationalisation of LCDproc.
>>
>> I'm a British English speaker with minimal second language skills so I
>> apologise that I have only addressed those areas that are relevant to
>> me. My problem was that I have a number of Celtic &  Gaelic tracks in
>> my music collection and the accented characters used in those
>> languages were not being displayed correctly.
>>
>> UTF-8 characters greater than 0x7F were displayed as two or more
>> strange characters:
>>
>>
>> With the patch and a display that supports ISO-8859-1 most western
>> European languages can be displayed:
>>
>>
>>
>> I've also added string support for Unicode on graphic displays:
>>
>>
>> I've not done title or scroller widgets because I feel graphic
>> displays should support pixel level scrolling and proportional spaced
>> fonts which would require more work than I am prepared for (I don't
>> have a graphic display). Basic string support was coded as proof of
>> concept and as a test case.
>>
>> There is obviously more to do but the code in the patch meets my
>> requirements and I hope will be of use to others.
>>
>> Further work I have thought of:
>>
>>    * Conversion of menu text
>>    * Use of gettext() and conversion for system screens
>>    * Use of gettext() and conversion for lcdproc screens
>>
>> Regards
>>
>> Martin
>
> _______________________________________________
> LCDproc mailing list
> [email protected]
> http://lists.omnipotent.net/mailman/listinfo/lcdproc


--------------020303080309060606010401
Content-Type: text/html; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit

<html>
  <head>
    <meta content="text/html; charset=ISO-8859-1"
      http-equiv="Content-Type">
  </head>
  <body bgcolor="#FFFFFF" text="#000000">
    <div class="moz-cite-prefix">On 06/03/14 07:00, Markus Dolze wrote:<br>
    </div>
    <blockquote cite="mid:[email protected]" type="cite">
      <pre wrap="">Hi,

character encoding - where should I begin?</pre>
    </blockquote>
    A big subject I agree, however I am trying to provide a solution to
    a relatively simple specific problem.<br>
    <blockquote cite="mid:[email protected]" type="cite">
      <pre wrap="">

It is true that LCDd does not handle anything else than single byte
characters as input from clients.

However, it does not assume any encoding - other than it contains
characters that can be parsed into commands.</pre>
    </blockquote>
    There are some implied assumptions:<br>
    <ol>
      <li>Commands are in ASCII so an encoding has to be a superset of
        ASCII (e.g. ISO/IEC 8859-n or UTF-8).</li>
      <li>Text to be displayed will map to the appropriate glyphs on the
        display.<br>
      </li>
    </ol>
    <blockquote cite="mid:[email protected]" type="cite">
      <pre wrap="">

Any data is passed to the drivers 'as is'. Some drivers map that (single
byte) characters to values in CGROM of the display. The Crystalfontz and
HD4480 drivers e.g. assume ISO 8859-1 as input by default and can handle
ISO 8859-5, CP-1251, KOI8-R as well.

The picolcd driver does not do any character mapping.


Currently I am fine with this, except the developer's guide does not
talk about it. Clients are responsible for sending single byte
characters to LCDd (whatever encoding).</pre>
    </blockquote>
    The graphic drivers have the capability to display any Unicode
    character so the input needs to move away from single byte
    characters. UTF-8 allows encoding any Unicode character into a byte
    stream.<br>
    <blockquote cite="mid:[email protected]" type="cite">
      <pre wrap="">

I am not agaist using UTF-8 as input. If we want to use it, my opinion
is not to apply it to some 'widget text value', but to the whole input
before it is parsed.</pre>
    </blockquote>
    I put the conversion on the widget text so that debug etc. reports
    the input as sent. It should work elsewhere if that is preferred.
    Conversion should have no effect on commands, they are ASCII so will
    not be changed for any encoding lightly to be used.<br>
    <blockquote cite="mid:[email protected]" type="cite">
      <pre wrap="">

By using libiconv we may get rid of mapping non ISO 8859-1 encoded input
to CGROM values. Or not.</pre>
    </blockquote>
    Depends how well the CGROM characters map to an encoding supported
    by the system, it may be possible to generate a mapping table for
    libiconv for each display.<br>
    <blockquote cite="mid:[email protected]" type="cite">
      <pre wrap="">

However, auto-detecting the system locale only works if client and
server run on the same system and share the same locale (which might not
be true for LCDd started at boot time and a client that runs in a user
session).</pre>
    </blockquote>
    I've allowed the locale to be set from the configuration, maybe the
    comment needs more explanation.<br>
    <blockquote cite="mid:[email protected]" type="cite">
      <pre wrap="">

Library dependencies and compatibility are another topic for some future
discussion.

Regards,
Markus


On 27.02.2014 21:47, Martin Tharby Jones wrote:
</pre>
      <blockquote type="cite">
        <pre wrap="">Hello,

I've attached a patch which provides a first step towards
internationalisation of LCDproc.

I'm a British English speaker with minimal second language skills so I
apologise that I have only addressed those areas that are relevant to
me. My problem was that I have a number of Celtic &amp;  Gaelic tracks in
my music collection and the accented characters used in those
languages were not being displayed correctly.

UTF-8 characters greater than 0x7F were displayed as two or more
strange characters:


With the patch and a display that supports ISO-8859-1 most western
European languages can be displayed:



I've also added string support for Unicode on graphic displays:


I've not done title or scroller widgets because I feel graphic
displays should support pixel level scrolling and proportional spaced
fonts which would require more work than I am prepared for (I don't
have a graphic display). Basic string support was coded as proof of
concept and as a test case.

There is obviously more to do but the code in the patch meets my
requirements and I hope will be of use to others.

Further work I have thought of:

  * Conversion of menu text
  * Use of gettext() and conversion for system screens
  * Use of gettext() and conversion for lcdproc screens

Regards

Martin
</pre>
      </blockquote>
      <pre wrap="">

_______________________________________________
LCDproc mailing list
<a class="moz-txt-link-abbreviated" href="mailto:[email protected]">[email protected]</a>
<a class="moz-txt-link-freetext" href="http://lists.omnipotent.net/mailman/listinfo/lcdproc">http://lists.omnipotent.net/mailman/listinfo/lcdproc</a>
</pre>
    </blockquote>
    <br>
  </body>
</html>

--------------020303080309060606010401--

--===============2222717750721890078==
Content-Type: text/plain; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Disposition: inline

_______________________________________________
LCDproc mailing list
[email protected]
http://lists.omnipotent.net/mailman/listinfo/lcdproc

--===============2222717750721890078==--