Re[2]: [gobo-eiffel] Gelex with Unicode
"Alexander Kogtenkov kwaxer-JGs/[email protected] [gobo-eiffel]" <[email protected]> Tue, 30 Jun 2020 11:07:50 +0300
| Newsgroups | gmane.comp.lang.eiffel.gobo.general |
|---|---|
| Message-ID | <[email protected]> |
--MMweIIl2RfwWjqKWkqcqHZcUJ-9XINYdjSH1l22
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable
Hello Eric,
=C2=A0
Yes, I understand most of the points you mention, and thank you for the add=
itional clarification in the two recent messages. The difficult part is to =
keep everything in mind to build the complete picture. That=E2=80=99s why I=
mentioned it would be great to have some tables that summarize all the det=
ails. I=E2=80=99ve prepared such tables myself and I=E2=80=99m attaching th=
em to the email. (In case they do not pass through because of the mailing l=
ist filters, I=E2=80=99ll send the tables in private.) I=E2=80=99d like to =
keep them for the future, so let me know, please, if you would like to incl=
ude them in the documentation, or if there is some other place to put them.
=C2=A0
From your messages I derive that if I want to use Unicode patterns (not UTF=
-8 sequences, but real Unicode code points), I=E2=80=99d better not use opt=
ion "utf8". According to the tables, in this case I have to use YY_UNICODE_=
FILE_BUFFER, nothing else would work for all valid inputs.
=C2=A0
The variant that I miss in this picture is a full-fledged Unicode scanner w=
ith a file buffer that reads ISO 8859-1 and/or UTF-8, but performs conversi=
on on the fly. I.e., we keep the efficient scanner with minimum lookup than=
ks to Unicode support (again, this is a great addition, I like it very much=
, both direct Unicode code points and fantastic {+} and {-} operators, than=
k you!), but use the file buffer that reads whatever it has and delivers Un=
icode code points regardless of the underlying encoding. I guess, this is a=
variant that implements `item_code` to return UTF-32, but leaves both `as_=
special` and `as_unicode_special` returning Void. Do you think it would be =
too inefficient?
=C2=A0
Side note. It looks like setting the default encoding does not affect wheth=
er BOM is processed or not. I.e., if the default encoding is set to ISO 885=
9-1, the buffers still read and interpret BOM as if no encoding was specifi=
ed. So, if a file in ISO 8859-1 starts with a BOM, it would be interpreted =
as UTF-8 with the BOM removed. Maybe, I=E2=80=99m wrong here, but this is w=
hat I inferred from the code. Whether this is "a bug or a feature", I do no=
t know, but this may need to be either fixed or documented.
=C2=A0
Best regards,
Alexander Kogtenkov
=C2=A0
=C2=A0=20
>Eric Bezault ericb-D6Qt/9opevxWk0Htik3J/[email protected] [gobo-eiffel] <[email protected]=
>:
>=C2=A0
>Here are a few more remarks:
>
>1. It does not matter whether the .l file has a BOM or not.
>=C2=A0=C2=A0=C2=A0=C2=A0For example a pattern containing "=E2=88=80" (with=
a BOM) and a
>=C2=A0=C2=A0=C2=A0=C2=A0pattern containing "\u2200" (with or without a BOM=
) will
>=C2=A0=C2=A0=C2=A0=C2=A0produce the same scanner and will match the same i=
nputs.
>=C2=A0=C2=A0=C2=A0=C2=A0It's like writing "=E2=88=80" or "%/0x2200/" in an=
Eiffel class.
>=C2=A0=C2=A0=C2=A0=C2=A0It's two different ways to write the same Eiffel s=
tring.
>
>2. A pattern containing "=C3=A9" is not the same as a pattern
>=C2=A0=C2=A0=C2=A0=C2=A0containing "\xc3\xa9" (unless you use the option u=
tf8.
>=C2=A0=C2=A0=C2=A0=C2=A0see below). The first one expects one character, t=
he
>=C2=A0=C2=A0=C2=A0=C2=A0second one expects two characters. The fact that t=
hey
>=C2=A0=C2=A0=C2=A0=C2=A0happen to be the bytes of the UTF-8 encoding of th=
e first
>=C2=A0=C2=A0=C2=A0=C2=A0one is irrelevant. They could have been the bytes =
of
>=C2=A0=C2=A0=C2=A0=C2=A0any other encoding. The scanner does not care. It =
just
>=C2=A0=C2=A0=C2=A0=C2=A0expects two characters.
>
>3. When using the option utf8, you tell gelex to replace
>=C2=A0=C2=A0=C2=A0=C2=A0characters in the pattern by their UTF-8 byte sequ=
ences.
>=C2=A0=C2=A0=C2=A0=C2=A0So if it finds a pattern "=C3=A9" it will treat it=
as if you
>=C2=A0=C2=A0=C2=A0=C2=A0had written "\xc3\xa9". So it really expects to ma=
tch
>=C2=A0=C2=A0=C2=A0=C2=A0these two characters, and will fail if it receives=
the
>=C2=A0=C2=A0=C2=A0=C2=A0single character "=C3=A9" instead.
>
>4. It does not matter whether the scanner receives the
>=C2=A0=C2=A0=C2=A0=C2=A0character {CHARACTER_8}'=C3=A9' or {CHARACTER_32}'=
=C3=A9'. It
>=C2=A0=C2=A0=C2=A0=C2=A0treats them the same because what counts is the ch=
aracter
>=C2=A0=C2=A0=C2=A0=C2=A0code, which is 233 in both cases.
>
>5. Writing a scanner which is able to match the UTF-8 byte
>=C2=A0=C2=A0=C2=A0=C2=A0sequence with the intend to avoid having to conver=
t back
>=C2=A0=C2=A0=C2=A0=C2=A0and forth from UTF-8 to STRING_32 is not a good id=
ea in
>=C2=A0=C2=A0=C2=A0=C2=A0most cases. It produces larger scanners (bigger in=
ternal
>=C2=A0=C2=A0=C2=A0=C2=A0tables), without the expected speed improvement be=
cause
>=C2=A0=C2=A0=C2=A0=C2=A0for a given Unicode character instead of having on=
e
>=C2=A0=C2=A0=C2=A0=C2=A0table lookup (or 2 or 3 lookups in case of a compr=
essed
>=C2=A0=C2=A0=C2=A0=C2=A0scanner) to see whether the input character matche=
s
>=C2=A0=C2=A0=C2=A0=C2=A0or not, you have to multiply that by the number of=
bytes
>=C2=A0=C2=A0=C2=A0=C2=A0corresponding to the UTF-8 encoding for this chara=
cter.
>=C2=A0=C2=A0=C2=A0=C2=A0So in the end the speed is about the same, but the=
size
>=C2=A0=C2=A0=C2=A0=C2=A0of the scanner is smaller.
>
>--
>Eric Bezault
>mailto:ericb-D6Qt/9opevxWk0Htik3J/[email protected]
>http://www.gobosoft.com
>
>
>On 30/06/2020 0:30, Eric Bezault ericb-D6Qt/9opevxWk0Htik3J/[email protected] [gobo-eiffel] wrote:
>> Hello Alexander,
>>
>> I'm sorry to hear that you have troubles using gelex. It has been
>> some times now since I worked on it, so I don't remember all the
>> details. But what I remember is that I worked hard to make gelex
>> work in a flexible way so that it could still work with existing
>> ..l files as well as with .l files containing Unicode characters.
>> And as far as I know, we can combine various input buffers. But
>> it's like any Eiffel program where you pass a STRING_8 to a
>> routine for example. If the routine expects this string to actually
>> contain UTF-8 bytes and you pass "=C3=A9" instead of the sequence
>> of two bytes corresponding to this character in UTF-8 encoding,
>> then the routine will not work as expected. But you don't get
>> a compilation error. Same here for gelex.
>>
>> But in fact things are simpler than they might look like after
>> reading your message below. So let me present it in a simple
>> way:
>>
>> * The scanner generated by gelex can accept CHARACTER_8 or
>> CHARACTER_32 sequences, regardless of whether the patterns
>> in the scanner are written with ISO 8859-1 characters or
>> Unicode characters.
>>
>> * The input buffer reads files and produce sequences of
>> characters (either CHARACTER_8 or CHARACTER_32) that
>> will be sent to the scanner.
>>
>> * YY_FILE_BUFFER reads bytes as they are in the input file
>> and produces a sequence of CHARACTER_8, one CHARACTER_8
>> per byte (including the bytes of the BOM if present).
>> This means that '=C3=A9' will be represented as a single
>> CHARACTER_8 character.
>>
>> * YY_UTF8_FILE_BUFFER can read files containing ISO 8859-1
>> characters (no BOM) or Unicode characters (with UTF-8 BOM).
>> It will produce a sequence of CHARACTER_8 corresponding
>> to the bytes of the UTF-8 representation of what has been
>> read. So if the input file was encoded in UTF-8, it will
>> skip the BOM and then send the bytes unchanged. If it
>> was ISO 8859-1, it will convert the characters to UTF-8
>> (e.g. the character '=C3=A9' will be replaced by its two byte
>> UTF-8 encoding).
>>
>> * YY_UNICODE_FILE_BUFFER can read files containing ISO 8859-1
>> characters (no BOM) or Unicode characters (with UTF-8 BOM).
>> It will produce a sequence of CHARACTER_32 characters.
>> So if the input file was encoded in UTF-8, it will replace
>> the UTF-8 byte sequences with their corresponding Unicode
>> characters. If it was ISO 8859-1, it will just convert the
>> characters to CHARACTER_32 (e.g. the character '=C3=A9' will
>> be converted to the character {CHARACTER_32}'=C3=A9').
>>
>> Now, you can write a scanner with patterns written in
>> ISO 8859-1 and use a YY_UNICODE_FILE_BUFFER. If will
>> work. Internally the scanner uses character codes
>> as indexes to internal tables. And since {CHARACTER_8}'=C3=A9'
>> and {CHARACTER_32}'=C3=A9' have the same code, it will
>> match. Of course if the input file contains Unicode
>> characters with code greater than 255, they will not
>> match the patterns in the .l file which only contains
>> characters with codes less than 255. And similarly,
>> you can write a scanner with patterns written in
>> Unicode and use YY_FILE_BUFFER. If the pattern is
>> of the form '[a=E2=88=80]' and the input is 'a' then it will
>> match. See the section "Some notes on patterns" in
>> http://www.gobosoft.com/eiffel/gobo/gelex/patterns.html
>> about the use of Unicode characters in patterns.
>> And of course, if you get a Unicode string which
>> cannot be represented as a STRING_8 and you call
>> `text` (instead of `unicode_text` or `utf8_text`) you get
>> some truncated results:
>> http://www.gobosoft.com/eiffel/gobo/gelex/actions.html#text
>>
>> So far, we had scanners:
>>
>> * which were expecting ISO 8859-1 input files and were using
>> YY_FILE_BUFFER
>> * which were expecting Unicode input files encoded in UTF-8
>> and which were using an equivalent of YY_UTF8_FILE_BUFFER
>> and using patterns containing the explicit UTF-8 byte
>> sequences (which is the case of the Eiffel scanner in
>> EiffelStudio:
>> https://github.com/EiffelSoftware/EiffelStudio/blob/master/Src/framewor=
k/parser/parser/eiffel.l
>> )
>>
>> and now we have scanners:
>>
>> * which are expecting Unicode input files encoded in UTF-8
>> and which are using YY_UNICODE_FILE_BUFFER and using
>> patterns containing Unicode characters (which is the case
>> of the Eiffel scanner in Gobo:
>> https://github.com/gobo-eiffel/gobo/blob/master/library/tools/src/eiffe=
l/parser/et_eiffel_scanner.l
>> )
>>
>>
>> But of course, if you write patterns containing Unicode characters
>> and you use a file buffer which sends UTF-8 bytes that will not
>> work because the bytes will be received as CHARACTER_8 and
>> interpreted as characters. It's like the example that I gave at the
>> beginning of this message where you have a routine with an
>> argument of type STRING_8 and it expects a UTF-8 sequence of bytes
>> and you pass "=C3=A9" (or vice-versa).
>>
>> And to complicate things, we also have the option utf8:
>> http://www.gobosoft.com/eiffel/gobo/gelex/options.html#utf8
>> Initially, the idea to support Unicode in gelex was to allow
>> patterns to be written with Unicode characters, but let gelex
>> convert them to UTF-8 byte sequences behind the scene, and
>> hence generate the same kind of scanner that is currently
>> used by EiffelStudio. This scanner expects a sequence of
>> UTF-8 bytes as input, e.g. YY_UTF8_FILE_BUFFER, or
>> YY_FILE_BUFFER if the input file only contains ASCII
>> characters. But I don't recommend using this mode because
>> in most cases it is not as good as using YY_UNICODE_FILE_BUFFER.
>> But I mention it here because this option is what you were
>> missing when you tried to use patterns containing Unicode
>> characters with a YY_UTF8_FILE_BUFFER.
>>
>> One last remark: the input buffers are not limited to
>> YY_FILE_BUFFER, YY_UTF8_FILE_BUFFER and YY_UNICODE_FILE_BUFFER.
>> One can write another class which would read files encoded
>> with UTF-16 and produce a sequence of CHARACTER_32 (or even
>> a sequence of UTF-8 bytes).
>>
>> And as a reminder, you can have a look at this small example:
>> https://github.com/gobo-eiffel/gobo/tree/master/library/lexical/example=
/unicode
>>
>
>
>
>
>
>------------------------------------
>Posted by: Eric Bezault < ericb-D6Qt/9opevxWk0Htik3J/[email protected] >
>------------------------------------
>
>To Post a message, send it to: [email protected]
>To Unsubscribe, send a blank message to: gobo-eiffel-unsubscribe@eGroups.=
com
>------------------------------------
>
>Yahoo Groups Links
>
>
>
=C2=A0
[Non-text portions of this message have been removed]
--MMweIIl2RfwWjqKWkqcqHZcUJ-9XINYdjSH1l22
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: quoted-printable
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN" "http://www.w3.org/TR/htm=
l4/strict.dtd">
<html>
<head>
</head>
=20
<body style=3D"background-color: #fff;">
<span style=3D"display:none"> </span>
<!--~-|**|PrettyHtmlStartT|**|-~-->
<div id=3D"ygrp-mlmsg" style=3D"position:relative;">
<div id=3D"ygrp-msg" style=3D"z-index: 1;">
<!--~-|**|PrettyHtmlEndT|**|-~-->
<div id=3D"ygrp-text" >
=20=20=20=20=20=20
=20=20=20=20=20=20
<p><br>
Hello Eric,<br>
=C2=A0<br>
Yes, I understand most of the points you mention, and thank you for the add=
itional clarification in the two recent messages. The difficult part is to =
keep everything in mind to build the complete picture. That=E2=80=99s why I=
mentioned it would be great to have some tables that summarize all the det=
ails. I=E2=80=99ve prepared such tables myself and I=E2=80=99m attaching th=
em to the email. (In case they do not pass through because of the mailing l=
ist filters, I=E2=80=99ll send the tables in private.) I=E2=80=99d like to =
keep them for the future, so let me know, please, if you would like to incl=
ude them in the documentation, or if there is some other place to put them.=
<br>
=C2=A0<br>
From your messages I derive that if I want to use Unicode patterns (not UTF=
-8 sequences, but real Unicode code points), I=E2=80=99d better not use opt=
ion "utf8". According to the tables, in this case I have to use Y=
Y_UNICODE_FILE_BUFFER, nothing else would work for all valid inputs.<br>
=C2=A0<br>
The variant that I miss in this picture is a full-fledged Unicode scanner w=
ith a file buffer that reads ISO 8859-1 and/or UTF-8, but performs conversi=
on on the fly. I.e., we keep the efficient scanner with minimum lookup than=
ks to Unicode support (again, this is a great addition, I like it very much=
, both direct Unicode code points and fantastic {+} and {-} operators, than=
k you!), but use the file buffer that reads whatever it has and delivers Un=
icode code points regardless of the underlying encoding. I guess, this is a=
variant that implements `item_code` to return UTF-32, but leaves both `as_=
special` and `as_unicode_special` returning Void. Do you think it would be =
too inefficient?<br>
=C2=A0<br>
Side note. It looks like setting the default encoding does not affect wheth=
er BOM is processed or not. I.e., if the default encoding is set to ISO 885=
9-1, the buffers still read and interpret BOM as if no encoding was specifi=
ed. So, if a file in ISO 8859-1 starts with a BOM, it would be interpreted =
as UTF-8 with the BOM removed. Maybe, I=E2=80=99m wrong here, but this is w=
hat I inferred from the code. Whether this is "a bug or a feature"=
;, I do not know, but this may need to be either fixed or documented.<br>
=C2=A0<br>
Best regards,<br>
Alexander Kogtenkov<br>
=C2=A0<br>
=C2=A0 <br>
>Eric Bezault ericb-D6Qt/9opevxWk0Htik3J/[email protected] [gobo-eiffel] <gobo-eiffel@yahoogrou=
ps.com>:<br>
>=C2=A0<br>
>Here are a few more remarks:<br>
><br>
>1. It does not matter whether the .l file has a BOM or not.<br>
>=C2=A0=C2=A0=C2=A0=C2=A0For example a pattern containing "=E2=88=
=80" (with a BOM) and a<br>
>=C2=A0=C2=A0=C2=A0=C2=A0pattern containing "\u2200" (with=
or without a BOM) will<br>
>=C2=A0=C2=A0=C2=A0=C2=A0produce the same scanner and will match the sam=
e inputs.<br>
>=C2=A0=C2=A0=C2=A0=C2=A0It's like writing "=E2=88=80" or =
"%/0x2200/" in an Eiffel class.<br>
>=C2=A0=C2=A0=C2=A0=C2=A0It's two different ways to write the same E=
iffel string.<br>
><br>
>2. A pattern containing "=C3=A9" is not the same as a pattern=
<br>
>=C2=A0=C2=A0=C2=A0=C2=A0containing "\xc3\xa9" (unless=
you use the option utf8.<br>
>=C2=A0=C2=A0=C2=A0=C2=A0see below). The first one expects one character=
, the<br>
>=C2=A0=C2=A0=C2=A0=C2=A0second one expects two characters. The fact tha=
t they<br>
>=C2=A0=C2=A0=C2=A0=C2=A0happen to be the bytes of the UTF-8 encoding of=
the first<br>
>=C2=A0=C2=A0=C2=A0=C2=A0one is irrelevant. They could have been the byt=
es of<br>
>=C2=A0=C2=A0=C2=A0=C2=A0any other encoding. The scanner does not care. =
It just<br>
>=C2=A0=C2=A0=C2=A0=C2=A0expects two characters.<br>
><br>
>3. When using the option utf8, you tell gelex to replace<br>
>=C2=A0=C2=A0=C2=A0=C2=A0characters in the pattern by their UTF-8 byte s=
equences.<br>
>=C2=A0=C2=A0=C2=A0=C2=A0So if it finds a pattern "=C3=A9" it =
will treat it as if you<br>
>=C2=A0=C2=A0=C2=A0=C2=A0had written "\xc3\xa9". So it=
really expects to match<br>
>=C2=A0=C2=A0=C2=A0=C2=A0these two characters, and will fail if it recei=
ves the<br>
>=C2=A0=C2=A0=C2=A0=C2=A0single character "=C3=A9" instead.<br=
>
><br>
>4. It does not matter whether the scanner receives the<br>
>=C2=A0=C2=A0=C2=A0=C2=A0character {CHARACTER_8}'=C3=A9' or {CHA=
RACTER_32}'=C3=A9'. It<br>
>=C2=A0=C2=A0=C2=A0=C2=A0treats them the same because what counts is the=
character<br>
>=C2=A0=C2=A0=C2=A0=C2=A0code, which is 233 in both cases.<br>
><br>
>5. Writing a scanner which is able to match the UTF-8 byte<br>
>=C2=A0=C2=A0=C2=A0=C2=A0sequence with the intend to avoid having to con=
vert back<br>
>=C2=A0=C2=A0=C2=A0=C2=A0and forth from UTF-8 to STRING_32 is not a good=
idea in<br>
>=C2=A0=C2=A0=C2=A0=C2=A0most cases. It produces larger scanners (bigger=
internal<br>
>=C2=A0=C2=A0=C2=A0=C2=A0tables), without the expected speed improvement=
because<br>
>=C2=A0=C2=A0=C2=A0=C2=A0for a given Unicode character instead of having=
one<br>
>=C2=A0=C2=A0=C2=A0=C2=A0table lookup (or 2 or 3 lookups in case of a co=
mpressed<br>
>=C2=A0=C2=A0=C2=A0=C2=A0scanner) to see whether the input character mat=
ches<br>
>=C2=A0=C2=A0=C2=A0=C2=A0or not, you have to multiply that by the number=
of bytes<br>
>=C2=A0=C2=A0=C2=A0=C2=A0corresponding to the UTF-8 encoding for this ch=
aracter.<br>
>=C2=A0=C2=A0=C2=A0=C2=A0So in the end the speed is about the same, but =
the size<br>
>=C2=A0=C2=A0=C2=A0=C2=A0of the scanner is smaller.<br>
><br>
>--<br>
>Eric Bezault<br>
>mailto:ericb-D6Qt/9opevxWk0Htik3J/[email protected]<br>
>http://www.gobosoft.com<br>
><br>
><br>
>On 30/06/2020 0:30, Eric Bezault ericb-D6Qt/9opevxWk0Htik3J/[email protected] [gobo-eiffel] wrot=
e:<br>
>> Hello Alexander,<br>
>><br>
>> I'm sorry to hear that you have troubles using gelex. It has b=
een<br>
>> some times now since I worked on it, so I don't remember all t=
he<br>
>> details. But what I remember is that I worked hard to make gelex<b=
r>
>> work in a flexible way so that it could still work with existing<b=
r>
>> ..l files as well as with .l files containing Unicode characters.<=
br>
>> And as far as I know, we can combine various input buffers. But<br=
>
>> it's like any Eiffel program where you pass a STRING_8 to a<br=
>
>> routine for example. If the routine expects this string to actuall=
y<br>
>> contain UTF-8 bytes and you pass "=C3=A9" instead of the=
sequence<br>
>> of two bytes corresponding to this character in UTF-8 encoding,<br=
>
>> then the routine will not work as expected. But you don't get<=
br>
>> a compilation error. Same here for gelex.<br>
>><br>
>> But in fact things are simpler than they might look like after<br>
>> reading your message below. So let me present it in a simple<br>
>> way:<br>
>><br>
>> * The scanner generated by gelex can accept CHARACTER_8 or<br>
>> CHARACTER_32 sequences, regardless of whether the patterns<br>
>> in the scanner are written with ISO 8859-1 characters or<br>
>> Unicode characters.<br>
>><br>
>> * The input buffer reads files and produce sequences of<br>
>> characters (either CHARACTER_8 or CHARACTER_32) that<br>
>> will be sent to the scanner.<br>
>><br>
>> * YY_FILE_BUFFER reads bytes as they are in the input file<br>
>> and produces a sequence of CHARACTER_8, one CHARACTER_8<br>
>> per byte (including the bytes of the BOM if present).<br>
>> This means that '=C3=A9' will be represented as a single<b=
r>
>> CHARACTER_8 character.<br>
>><br>
>> * YY_UTF8_FILE_BUFFER can read files containing ISO 8859-1<br>
>> characters (no BOM) or Unicode characters (with UTF-8 BOM).<br>
>> It will produce a sequence of CHARACTER_8 corresponding<br>
>> to the bytes of the UTF-8 representation of what has been<br>
>> read. So if the input file was encoded in UTF-8, it will<br>
>> skip the BOM and then send the bytes unchanged. If it<br>
>> was ISO 8859-1, it will convert the characters to UTF-8<br>
>> (e.g. the character '=C3=A9' will be replaced by its two b=
yte<br>
>> UTF-8 encoding).<br>
>><br>
>> * YY_UNICODE_FILE_BUFFER can read files containing ISO 8859-1<br>
>> characters (no BOM) or Unicode characters (with UTF-8 BOM).<br>
>> It will produce a sequence of CHARACTER_32 characters.<br>
>> So if the input file was encoded in UTF-8, it will replace<br>
>> the UTF-8 byte sequences with their corresponding Unicode<br>
>> characters. If it was ISO 8859-1, it will just convert the<br>
>> characters to CHARACTER_32 (e.g. the character '=C3=A9' wi=
ll<br>
>> be converted to the character {CHARACTER_32}'=C3=A9').<br>
>><br>
>> Now, you can write a scanner with patterns written in<br>
>> ISO 8859-1 and use a YY_UNICODE_FILE_BUFFER. If will<br>
>> work. Internally the scanner uses character codes<br>
>> as indexes to internal tables. And since {CHARACTER_8}'=C3=A9&=
#39;<br>
>> and {CHARACTER_32}'=C3=A9' have the same code, it will<br>
>> match. Of course if the input file contains Unicode<br>
>> characters with code greater than 255, they will not<br>
>> match the patterns in the .l file which only contains<br>
>> characters with codes less than 255. And similarly,<br>
>> you can write a scanner with patterns written in<br>
>> Unicode and use YY_FILE_BUFFER. If the pattern is<br>
>> of the form '[a=E2=88=80]' and the input is 'a' th=
en it will<br>
>> match. See the section "Some notes on patterns" in<br>
>> http://www.gobosoft.com/eiffel/gobo/gelex/patterns.html<br>
>> about the use of Unicode characters in patterns.<br>
>> And of course, if you get a Unicode string which<br>
>> cannot be represented as a STRING_8 and you call<br>
>> `text` (instead of `unicode_text` or `utf8_text`) you get<br>
>> some truncated results:<br>
>> http://www.gobosoft.com/eiffel/gobo/gelex/actions.html#text<br>
>><br>
>> So far, we had scanners:<br>
>><br>
>> * which were expecting ISO 8859-1 input files and were using<br>
>> YY_FILE_BUFFER<br>
>> * which were expecting Unicode input files encoded in UTF-8<br>
>> and which were using an equivalent of YY_UTF8_FILE_BUFFER<br>
>> and using patterns containing the explicit UTF-8 byte<br>
>> sequences (which is the case of the Eiffel scanner in<br>
>> EiffelStudio:<br>
>> https://github.com/EiffelSoftware/EiffelStudio/blob/master/Src/fr=
amework/parser/parser/eiffel.l<br>
>> )<br>
>><br>
>> and now we have scanners:<br>
>><br>
>> * which are expecting Unicode input files encoded in UTF-8<br>
>> and which are using YY_UNICODE_FILE_BUFFER and using<br>
>> patterns containing Unicode characters (which is the case<br>
>> of the Eiffel scanner in Gobo:<br>
>> https://github.com/gobo-eiffel/gobo/blob/master/library/tools/src=
/eiffel/parser/et_eiffel_scanner.l<br>
>> )<br>
>><br>
>><br>
>> But of course, if you write patterns containing Unicode characters=
<br>
>> and you use a file buffer which sends UTF-8 bytes that will not<br=
>
>> work because the bytes will be received as CHARACTER_8 and<br>
>> interpreted as characters. It's like the example that I gave a=
t the<br>
>> beginning of this message where you have a routine with an<br>
>> argument of type STRING_8 and it expects a UTF-8 sequence of bytes=
<br>
>> and you pass "=C3=A9" (or vice-versa).<br>
>><br>
>> And to complicate things, we also have the option utf8:<br>
>> http://www.gobosoft.com/eiffel/gobo/gelex/options.html#utf8<br>
>> Initially, the idea to support Unicode in gelex was to allow<br>
>> patterns to be written with Unicode characters, but let gelex<br>
>> convert them to UTF-8 byte sequences behind the scene, and<br>
>> hence generate the same kind of scanner that is currently<br>
>> used by EiffelStudio. This scanner expects a sequence of<br>
>> UTF-8 bytes as input, e.g. YY_UTF8_FILE_BUFFER, or<br>
>> YY_FILE_BUFFER if the input file only contains ASCII<br>
>> characters. But I don't recommend using this mode because<br>
>> in most cases it is not as good as using YY_UNICODE_FILE_BUFFER.<b=
r>
>> But I mention it here because this option is what you were<br>
>> missing when you tried to use patterns containing Unicode<br>
>> characters with a YY_UTF8_FILE_BUFFER.<br>
>><br>
>> One last remark: the input buffers are not limited to<br>
>> YY_FILE_BUFFER, YY_UTF8_FILE_BUFFER and YY_UNICODE_FILE_BUFFER.<br=
>
>> One can write another class which would read files encoded<br>
>> with UTF-16 and produce a sequence of CHARACTER_32 (or even<br>
>> a sequence of UTF-8 bytes).<br>
>><br>
>> And as a reminder, you can have a look at this small example:<br>
>> https://github.com/gobo-eiffel/gobo/tree/master/library/lexical/e=
xample/unicode<br>
>><br>
><br>
><br>
><br>
><br>
><br>
>------------------------------------<br>
>Posted by: Eric Bezault < ericb-D6Qt/9opevxWk0Htik3J/[email protected] ><br>
>------------------------------------<br>
><br>
>To Post a message, send it to: [email protected]<br>
>To Unsubscribe, send a blank message to: gobo-eiffel-unsubscribe@eGrou=
ps.com<br>
>------------------------------------<br>
><br>
>Yahoo Groups Links<br>
><br>
><br>
><br>
=C2=A0<br>
<br>
[Non-text portions of this message have been removed]<br>
<br>
</p>
</div>
=20=20=20=20=20
<!--~-|**|PrettyHtmlStart|**|-~-->
<div style=3D"color: #fff; height: 0;">__._,_.___</div>
=20=20=20=20=20=20=20=20=20=20
=20=20
=20
=20=20=20=20
<div style=3D"clear:both"> </div>
<div id=3D"fromDMARC" style=3D"margin-top: 10px;">
<hr style=3D"height:2px ; border-width:0; color:#E3E3E3; background=
-color:#E3E3E3;">
Posted by: =3D?UTF-8?B?QWxleGFuZGVyIEtvZ3Rlbmtvdg=3D=3D?=3D <kwa=
xer-JGs/[email protected]> <hr style=3D"height:2px ; border-width:0; color:#E3E=
3E3; background-color:#E3E3E3;">
</div>
<div style=3D"clear:both"> </div>
<table cellspacing=3D4px style=3D"margin-top: 10px; margin-bottom: 10px=
; color: #2D50FD;">
<tbody>
<tr>
<td style=3D"font-size: 12px; font-family: arial; font-weight: bo=
ld; padding: 7px 5px 5px;" >
<a style=3D"text-decoration: none; color: #2D50FD=
" href=3D"https://groups.yahoo.com/neo/groups/gobo-eiffel/conversations/mes=
sages/1926;_ylc=3DX3oDMTJwcm84M2lzBF9TAzk3MzU5NzE0BGdycElkAzE1MjIyNzAEZ3Jwc=
3BJZAMxNzA1MDA2NzY0BG1zZ0lkAzE5MjYEc2VjA2Z0cgRzbGsDcnBseQRzdGltZQMxNTkzNTA0=
NDc1?act=3Dreply&messageNum=3D1926">Reply via web post</a>
</td>
<td>•</td>
<td style=3D"font-size: 12px; font-family: arial; padding: 7px 5p=
x 5px;" >
<a href=3D"mailto:kwaxer-JGs/[email protected]?subject=3DRe%3A%20Re%5B2%5D%3A=
%20%5Bgobo-eiffel%5D%20Gelex%20with%20Unicode" style=3D"text-decoration: no=
ne; color: #2D50FD;">
Reply to sender </a>
</td>
<td>•</td>
<td style=3D"font-size: 12px; font-family: arial; padding: 7px 5p=
x 5px;">
<a href=3D"mailto:[email protected]?subject=3DRe%3A%2=
0Re%5B2%5D%3A%20%5Bgobo-eiffel%5D%20Gelex%20with%20Unicode" style=3D"text-d=
ecoration: none; color: #2D50FD">
Reply to group </a>
</td>
<td>•</td>
<td style=3D"font-size: 12px; font-family: arial; padding: 7px 5p=
x 5px;" >
<a href=3D"https://groups.yahoo.com/neo/groups/gobo-eiffel/conv=
ersations/newtopic;_ylc=3DX3oDMTJlMXFzNWozBF9TAzk3MzU5NzE0BGdycElkAzE1MjIyN=
zAEZ3Jwc3BJZAMxNzA1MDA2NzY0BHNlYwNmdHIEc2xrA250cGMEc3RpbWUDMTU5MzUwNDQ3NQ--=
" style=3D"text-decoration: none; color: #2D50FD">Start a New Topic</a>
</td>
<td>•</td>
<td style=3D"font-size: 12px; font-family: arial; padding: 7px 5p=
x 5px;color: #2D50FD;" >
<a href=3D"https://groups.yahoo.com/neo/groups/=
gobo-eiffel/conversations/topics/1923;_ylc=3DX3oDMTM0MHVoY3RrBF9TAzk3MzU5Nz=
E0BGdycElkAzE1MjIyNzAEZ3Jwc3BJZAMxNzA1MDA2NzY0BG1zZ0lkAzE5MjYEc2VjA2Z0cgRzb=
GsDdnRwYwRzdGltZQMxNTkzNTA0NDc1BHRwY0lkAzE5MjM-" style=3D"text-decoration: =
none; color: #2D50FD;">Messages in this topic</a>
(4)
</td>
</tr>
</tbody>
</table>
=20=20=20=20=20=20=20=20
<!------- Start Nav Bar ------>
<!-- |**|begin egp html banner|**| -->
<!-- |**|end egp html banner|**| -->
<div id=3D"ygrp-grfd" style=3D"font-family: Verdana; font-size: 12px; p=
adding: 15px 0;">
=20=20=20=20=20=20
<!-- |**|begin egp html banner|**| -->
To Post a message, send it to: [email protected]<BR=
>
To Unsubscribe, send a blank message to: gobo-eiffel-unsubscribe-B11MqFFcr06IwRZHo2/[email protected]=
m=20=20=20=20=20=20
<!-- |**|end egp html banner|**| -->
</div>
=20=20
=20
<!-- |**|begin egp html banner|**| -->
<div id=3D"ygrp-vital" style=3D"background-color: #f2f2f2; font-family: Ver=
dana; font-size: 10px; margin-bottom: 10px; padding: 10px;">
<span id=3D"vithd" style=3D"font-weight: bold; color: #333; text-transf=
orm: uppercase; "><a href=3D"https://groups.yahoo.com/neo/groups/gobo-eiffe=
l/info;_ylc=3DX3oDMTJlam9tYWllBF9TAzk3MzU5NzE0BGdycElkAzE1MjIyNzAEZ3Jwc3BJZ=
AMxNzA1MDA2NzY0BHNlYwN2dGwEc2xrA3ZnaHAEc3RpbWUDMTU5MzUwNDQ3NQ--" style=3D"t=
ext-decoration: none;">Visit Your Group</a></span>
<ul style=3D"list-style-type: none; margin: 0; padding: 0; display: in=
line;">
</ul>
</div>
<div id=3D"ft" style=3D"font-family: Arial; font-size: 11px; margin-top: 5p=
x; padding: 0 2px 0 0; clear: both;">
<a href=3D"https://groups.yahoo.com/neo;_ylc=3DX3oDMTJkdmNxMWg5BF9TAzk3Mz=
U5NzE0BGdycElkAzE1MjIyNzAEZ3Jwc3BJZAMxNzA1MDA2NzY0BHNlYwNmdHIEc2xrA2dmcARzd=
GltZQMxNTkzNTA0NDc1" style=3D"float: left;"><img src=3D"https://s.yimg.com/=
ru/0.9.12/min/css/yahoo_en-US_f_pw_101x21.png" height=3D"21" width=3D"101" =
alt=3D"Yahoo! Groups" style=3D"border: 0;"/></a>
<div style=3D"color: #747575; float: right;"> • <a href=3D"https://i=
nfo.yahoo.com/privacy/us/yahoo/groups/details.html" style=3D"text-decoratio=
n: none;">Privacy</a> • <a href=3D"mailto:gobo-eiffel-unsubscribe@yaho=
ogroups.com?subject=3DUnsubscribe" style=3D"text-decoration: none;">Unsubsc=
ribe</a> • <a href=3D"https://info.yahoo.com/legal/us/yahoo/utos/terms=
/" style=3D"text-decoration: none;">Terms of Use</a> </div>
</div>
<br>
<!-- |**|end egp html banner|**| -->
</div> <!-- ygrp-msg -->
=20
<!-- Sponsor -->
<!-- |**|begin egp html banner|**| -->
<div id=3D"ygrp-sponsor" style=3D"width:160px; float:right; clear:none; m=
argin:0 0 25px 0; background: #fff;">
<!-- Start Recommendations -->
<div id=3D"ygrp-reco">
</div>
<!-- End Recommendations -->
</div> <!-- |**|end egp html banner|**| -->
<div style=3D"clear:both; color: #FFF; font-size:1px;">.</div>
</div>
<img src=3D"http://geo.yahoo.com/serv?s=3D97359714/grpId=3D1522270/grpspI=
d=3D1705006764/msgId=3D1926/stime=3D1593504475" width=3D"1" height=3D"1"> <=
br>
<img src=3D"http://y.analytics.yahoo.com/fpc.pl?ywarid=3D515FB27823A7407E&a=
=3D10001310322279&js=3Dno&resp=3Dimg" width=3D"1" height=3D"1">=20
<div style=3D"color: #fff; height: 0;">__,_._,___</div>
<!--~-|**|PrettyHtmlEnd|**|-~-->
</body>
<!--~-|**|PrettyHtmlStart|**|-~-->
<head>
<style type=3D"text/css">
<!--
#ygrp-mkp {
border: 1px solid #d8d8d8;
font-family: Arial;
margin: 10px 0;
padding: 0 10px;
}
#ygrp-mkp hr {
border: 1px solid #d8d8d8;
}
#ygrp-mkp #hd {
color: #628c2a;
font-size: 85%;
font-weight: 700;
line-height: 122%;
margin: 10px 0;
}
#ygrp-mkp #ads {
margin-bottom: 10px;
}
#ygrp-mkp .ad {
padding: 0 0;
}
#ygrp-mkp .ad p {
margin: 0;
}
#ygrp-mkp .ad a {
color: #0000ff;
text-decoration: none;
}
#ygrp-sponsor #ygrp-lc {
font-family: Arial;
}
#ygrp-sponsor #ygrp-lc #hd {
margin: 10px 0px;
font-weight: 700;
font-size: 78%;
line-height: 122%;
}
#ygrp-sponsor #ygrp-lc .ad {
margin-bottom: 10px;
padding: 0 0;
}
#actions {
font-family: Verdana;
font-size: 11px;
padding: 10px 0;
}
#activity {
background-color: #e0ecee;
float: left;
font-family: Verdana;
font-size: 10px;
padding: 10px;
}
#activity span {
font-weight: 700;
}
#activity span:first-child {
text-transform: uppercase;
}
#activity span a {
color: #5085b6;
text-decoration: none;
}
#activity span span {
color: #ff7900;
}
#activity span .underline {
text-decoration: underline;
}
.attach {
clear: both;
display: table;
font-family: Arial;
font-size: 12px;
padding: 10px 0;
width: 400px;
}
.attach div a {
text-decoration: none;
}
.attach img {
border: none;
padding-right: 5px;
}
.attach label {
display: block;
margin-bottom: 5px;
}
.attach label a {
text-decoration: none;
}
=20=20
blockquote {
margin: 0 0 0 4px;
}
.bold {
font-family: Arial;
font-size: 13px;
font-weight: 700;
}
.bold a {
text-decoration: none;
}
dd.last p a {
font-family: Verdana;
font-weight: 700;
}
dd.last p span {
margin-right: 10px;
font-family: Verdana;
font-weight: 700;
}
dd.last p span.yshortcuts {
margin-right: 0;
}
div.attach-table div div a {
text-decoration: none;
}
div.attach-table {
width: 400px;
}
div.file-title a, div.file-title a:active, div.file-title a:hover, div.fi=
le-title a:visited {
text-decoration: none;
}
div.photo-title a, div.photo-title a:active, div.photo-title a:hover, div=
..photo-title a:visited {
text-decoration: none;
}
div#ygrp-mlmsg #ygrp-msg p a span.yshortcuts {
font-family: Verdana;
font-size: 10px;
font-weight: normal;
}
.green {
color: #628c2a;
}
.MsoNormal {
margin: 0 0 0 0;
}
o {
font-size: 0;
}
#photos div {
float: left;
width: 72px;
}
#photos div div {
border: 1px solid #666666;
height: 62px;
overflow: hidden;
width: 62px;
}
#photos div label {
color: #666666;
font-size: 10px;
overflow: hidden;
text-align: center;
white-space: nowrap;
width: 64px;
}
#reco-category {
font-size: 77%;
}
#reco-desc {
font-size: 77%;
}
.replbq {
margin: 4px;
}
#ygrp-actbar div a:first-child {
/* border-right: 0px solid #000;*/
margin-right: 2px;
padding-right: 5px;
}
#ygrp-mlmsg {
font-size: 13px;
font-family: Arial, helvetica,clean, sans-serif;
*font-size: small;
*font: x-small;
}
#ygrp-mlmsg table {
font-size: inherit;
font: 100%;
}
#ygrp-mlmsg select, input, textarea {
font: 99% Arial, Helvetica, clean, sans-serif;
}
#ygrp-mlmsg pre, code {
font:115% monospace;
*font-size:100%;
}
#ygrp-mlmsg * {
line-height: 1.22em;
}
#ygrp-mlmsg #logo {
padding-bottom: 10px;
}
#ygrp-msg p a {
font-family: Verdana;
}
#ygrp-msg p#attach-count span {
color: #1E66AE;
font-weight: 700;
}
#ygrp-reco #reco-head {
color: #ff7900;
font-weight: 700;
}
#ygrp-reco {
margin-bottom: 20px;
padding: 0px;
}
#ygrp-sponsor #ov li a {
font-size: 130%;
text-decoration: none;
}
#ygrp-sponsor #ov li {
font-size: 77%;
list-style-type: square;
padding: 6px 0;
}=20
#ygrp-sponsor #ov ul {
margin: 0;
padding: 0 0 0 8px;
}
#ygrp-text {
font-family: Georgia;
}
#ygrp-text p {
margin: 0 0 1em 0;
}
#ygrp-text tt {
font-size: 120%;
}
#ygrp-vital ul li:last-child {
border-right: none !important;=20
}=20
-->
</style>
</head>
<!--~-|**|PrettyHtmlEnd|**|-~-->
</html>
<!-- end group email -->
--MMweIIl2RfwWjqKWkqcqHZcUJ-9XINYdjSH1l22--