Re: Gelex with Unicode
"Eric Bezault ericb-D6Qt/9opevxWk0Htik3J/[email protected] [gobo-eiffel]" <[email protected]> Tue, 30 Jun 2020 10:46:31 +0200
| Newsgroups | gmane.comp.lang.eiffel.gobo.general |
|---|---|
| Message-ID | <[email protected]> |
--Ov15EKkbhJrWkjqP37tT1iYZjJ5FqltEevItD0S Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit On 30/06/2020 10:07, Alexander Kogtenkov kwaxer-JGs/[email protected] [gobo-eiffel] wrote: > Yes, I understand most of the points you mention, and thank you for the additional clarification in the two recent messages. The difficult part is to keep everything in mind to build the complete picture. That’s why I mentioned it would be great to have some tables that summarize all the details. I’ve prepared such tables myself and I’m attaching them to the email. (In case they do not pass through because of the mailing list filters, I’ll send the tables in private.) I’d like to keep them for the future, so let me know, please, if you would like to include them in the documentation, or if there is some other place to put them. I will have to study your table in details to validate whether it is accurate or not, but yes it's a good idea to add it to the documentation. >>From your messages I derive that if I want to use Unicode patterns (not UTF-8 sequences, but real Unicode code points), I’d better not use option "utf8". Yes. > According to the tables, in this case I have to use YY_UNICODE_FILE_BUFFER, nothing else would work for all valid inputs. YY_FILE_BUFFER would work if you know in advance that all input files are ISO 8859-1. But if you use Unicode patterns it is very likely that some, if not all, of your input files will contain Unicode characters. > The variant that I miss in this picture is a full-fledged Unicode scanner with a file buffer that reads ISO 8859-1 and/or UTF-8, but performs conversion on the fly. I.e., we keep the efficient scanner with minimum lookup thanks to Unicode support (again, this is a great addition, I like it very much, both direct Unicode code points and fantastic {+} and {-} operators, thank you!), but use the file buffer that reads whatever it has and delivers Unicode code points regardless of the underlying encoding. I'm not sure I understand what you mean. Isn't it what YY_UNICODE_FILE_BUFFER provides? Or do you mean that in addition to ISO 8859-1 and UTF-8, YY_UNICODE_FILE_BUFFER should also detect other encodings like UTF-16? > Side note. It looks like setting the default encoding does not affect whether BOM is processed or not. I.e., if the default encoding is set to ISO 8859-1, the buffers still read and interpret BOM as if no encoding was specified. So, if a file in ISO 8859-1 starts with a BOM, it would be interpreted as UTF-8 with the BOM removed. Maybe, I’m wrong here, but this is what I inferred from the code. Whether this is "a bug or a feature", I do not know, but this may need to be either fixed or documented. If I remember currently the default encoding is used when no BOM is found in the file. It does not override a BOM if found in the file. -- Eric Bezault mailto:ericb-D6Qt/9opevxWk0Htik3J/[email protected] http://www.gobosoft.com --Ov15EKkbhJrWkjqP37tT1iYZjJ5FqltEevItD0S Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN" "http://www.w3.org/TR/htm= l4/strict.dtd"> <html> <head> </head> =20 <body style=3D"background-color: #fff;"> <span style=3D"display:none"> </span> <!--~-|**|PrettyHtmlStartT|**|-~--> <div id=3D"ygrp-mlmsg" style=3D"position:relative;"> <div id=3D"ygrp-msg" style=3D"z-index: 1;"> <!--~-|**|PrettyHtmlEndT|**|-~--> <div id=3D"ygrp-text" > =20=20=20=20=20=20 =20=20=20=20=20=20 <p>On 30/06/2020 10:07, Alexander Kogtenkov kwaxer-JGs/[email protected] [gobo-eiff= el] <br> wrote: > Yes, I understand most of the points you mention, and thank <b= r> you for the additional clarification in the two recent messages. The <br> difficult part is to keep everything in mind to build the complete <br> picture. That=E2=80=99s why I mentioned it would be great to have some tabl= es <br> that summarize all the details. I=E2=80=99ve prepared such tables myself an= d I=E2=80=99m <br> attaching them to the email. (In case they do not pass through because <br> of the mailing list filters, I=E2=80=99ll send the tables in private.) I=E2= =80=99d like <br> to keep them for the future, so let me know, please, if you would like <br> to include them in the documentation, or if there is some other place to <b= r> put them.<br> I will have to study your table in details to validate whether it<br> is accurate or not, but yes it's a good idea to add it to the<br> documentation.<br> <br> >>From your messages I derive that if I want to use Unicode patterns = (not UTF-8 sequences, but real Unicode code points), I=E2=80=99d better not= use option "utf8".<br> <br> Yes.<br> <br> > According to the tables, in this case I have to use YY_UNICODE_FILE_BU= FFER, nothing else would work for all valid inputs.<br> <br> YY_FILE_BUFFER would work if you know in advance that all input files<br> are ISO 8859-1. But if you use Unicode patterns it is very likely that<br> some, if not all, of your input files will contain Unicode characters.<br> <br> > The variant that I miss in this picture is a full-fledged Unicode scan= ner with a file buffer that reads ISO 8859-1 and/or UTF-8, but performs con= version on the fly. I.e., we keep the efficient scanner with minimum lookup= thanks to Unicode support (again, this is a great addition, I like it very= much, both direct Unicode code points and fantastic {+} and {-} operators,= thank you!), but use the file buffer that reads whatever it has and delive= rs Unicode code points regardless of the underlying encoding.<br> <br> I'm not sure I understand what you mean. Isn't it what<br> YY_UNICODE_FILE_BUFFER provides? Or do you mean that in addition<br> to ISO 8859-1 and UTF-8, YY_UNICODE_FILE_BUFFER should also<br> detect other encodings like UTF-16?<br> <br> > Side note. It looks like setting the default encoding does not affect = whether BOM is processed or not. I.e., if the default encoding is set to IS= O 8859-1, the buffers still read and interpret BOM as if no encoding was sp= ecified. So, if a file in ISO 8859-1 starts with a BOM, it would be interpr= eted as UTF-8 with the BOM removed. Maybe, I=E2=80=99m wrong here, but this= is what I inferred from the code. Whether this is "a bug or a feature= ", I do not know, but this may need to be either fixed or documented.<= br> <br> If I remember currently the default encoding is used when no BOM is<br> found in the file. It does not override a BOM if found in the file.<br> <br> -- <br> Eric Bezault<br> mailto:ericb-D6Qt/9opevxWk0Htik3J/[email protected]<br> http://www.gobosoft.com<br> </p> </div> =20=20=20=20=20 <!--~-|**|PrettyHtmlStart|**|-~--> <div style=3D"color: #fff; height: 0;">__._,_.___</div> =20=20=20=20=20=20=20=20=20=20 =20=20 =20 =20=20=20=20 <div style=3D"clear:both"> </div> <div id=3D"fromDMARC" style=3D"margin-top: 10px;"> <hr style=3D"height:2px ; border-width:0; color:#E3E3E3; background= -color:#E3E3E3;"> Posted by: Eric Bezault <ericb-D6Qt/9opevxWk0Htik3J/[email protected]> <hr style= =3D"height:2px ; border-width:0; color:#E3E3E3; background-color:#E3E3E3;"> </div> <div style=3D"clear:both"> </div> <table cellspacing=3D4px style=3D"margin-top: 10px; margin-bottom: 10px= ; color: #2D50FD;"> <tbody> <tr> <td style=3D"font-size: 12px; font-family: arial; font-weight: bo= ld; padding: 7px 5px 5px;" > <a style=3D"text-decoration: none; color: #2D50FD= " href=3D"https://groups.yahoo.com/neo/groups/gobo-eiffel/conversations/mes= sages/1927;_ylc=3DX3oDMTJwaDlhZjJwBF9TAzk3MzU5NzE0BGdycElkAzE1MjIyNzAEZ3Jwc= 3BJZAMxNzA1MDA2NzY0BG1zZ0lkAzE5MjcEc2VjA2Z0cgRzbGsDcnBseQRzdGltZQMxNTkzNTA2= Nzk1?act=3Dreply&messageNum=3D1927">Reply via web post</a> </td> <td>•</td> <td style=3D"font-size: 12px; font-family: arial; padding: 7px 5p= x 5px;" > <a href=3D"mailto:ericb-D6Qt/9opevxWk0Htik3J/[email protected]?subject=3DRe%3A%20%5Bgobo-= eiffel%5D%20Gelex%20with%20Unicode" style=3D"text-decoration: none; color: = #2D50FD;"> Reply to sender </a> </td> <td>•</td> <td style=3D"font-size: 12px; font-family: arial; padding: 7px 5p= x 5px;"> <a href=3D"mailto:[email protected]?subject=3DRe%3A%2= 0%5Bgobo-eiffel%5D%20Gelex%20with%20Unicode" style=3D"text-decoration: none= ; color: #2D50FD"> Reply to group </a> </td> <td>•</td> <td style=3D"font-size: 12px; font-family: arial; padding: 7px 5p= x 5px;" > <a href=3D"https://groups.yahoo.com/neo/groups/gobo-eiffel/conv= ersations/newtopic;_ylc=3DX3oDMTJldjI0amNwBF9TAzk3MzU5NzE0BGdycElkAzE1MjIyN= zAEZ3Jwc3BJZAMxNzA1MDA2NzY0BHNlYwNmdHIEc2xrA250cGMEc3RpbWUDMTU5MzUwNjc5NQ--= " style=3D"text-decoration: none; color: #2D50FD">Start a New Topic</a> </td> <td>•</td> <td style=3D"font-size: 12px; font-family: arial; padding: 7px 5p= x 5px;color: #2D50FD;" > <a href=3D"https://groups.yahoo.com/neo/groups/= gobo-eiffel/conversations/topics/1923;_ylc=3DX3oDMTM0ZHJyYjl2BF9TAzk3MzU5Nz= E0BGdycElkAzE1MjIyNzAEZ3Jwc3BJZAMxNzA1MDA2NzY0BG1zZ0lkAzE5MjcEc2VjA2Z0cgRzb= GsDdnRwYwRzdGltZQMxNTkzNTA2Nzk1BHRwY0lkAzE5MjM-" style=3D"text-decoration: = none; color: #2D50FD;">Messages in this topic</a> (5) </td> </tr> </tbody> </table> =20=20=20=20=20=20=20=20 <!------- Start Nav Bar ------> <!-- |**|begin egp html banner|**| --> <!-- |**|end egp html banner|**| --> <div id=3D"ygrp-grfd" style=3D"font-family: Verdana; font-size: 12px; p= adding: 15px 0;"> =20=20=20=20=20=20 <!-- |**|begin egp html banner|**| --> To Post a message, send it to: [email protected]<BR= > To Unsubscribe, send a blank message to: gobo-eiffel-unsubscribe-B11MqFFcr06IwRZHo2/[email protected]= m=20=20=20=20=20=20 <!-- |**|end egp html banner|**| --> </div> =20=20 =20 <!-- |**|begin egp html banner|**| --> <div id=3D"ygrp-vital" style=3D"background-color: #f2f2f2; font-family: Ver= dana; font-size: 10px; margin-bottom: 10px; padding: 10px;"> <span id=3D"vithd" style=3D"font-weight: bold; color: #333; text-transf= orm: uppercase; "><a href=3D"https://groups.yahoo.com/neo/groups/gobo-eiffe= l/info;_ylc=3DX3oDMTJldmIxZThnBF9TAzk3MzU5NzE0BGdycElkAzE1MjIyNzAEZ3Jwc3BJZ= AMxNzA1MDA2NzY0BHNlYwN2dGwEc2xrA3ZnaHAEc3RpbWUDMTU5MzUwNjc5NQ--" style=3D"t= ext-decoration: none;">Visit Your Group</a></span> <ul style=3D"list-style-type: none; margin: 0; padding: 0; display: in= line;"> </ul> </div> <div id=3D"ft" style=3D"font-family: Arial; font-size: 11px; margin-top: 5p= x; padding: 0 2px 0 0; clear: both;"> <a href=3D"https://groups.yahoo.com/neo;_ylc=3DX3oDMTJkNDFwMXRiBF9TAzk3Mz= U5NzE0BGdycElkAzE1MjIyNzAEZ3Jwc3BJZAMxNzA1MDA2NzY0BHNlYwNmdHIEc2xrA2dmcARzd= GltZQMxNTkzNTA2Nzk1" style=3D"float: left;"><img src=3D"https://s.yimg.com/= ru/0.9.12/min/css/yahoo_en-US_f_pw_101x21.png" height=3D"21" width=3D"101" = alt=3D"Yahoo! Groups" style=3D"border: 0;"/></a> <div style=3D"color: #747575; float: right;"> • <a href=3D"https://i= nfo.yahoo.com/privacy/us/yahoo/groups/details.html" style=3D"text-decoratio= n: none;">Privacy</a> • <a href=3D"mailto:gobo-eiffel-unsubscribe@yaho= ogroups.com?subject=3DUnsubscribe" style=3D"text-decoration: none;">Unsubsc= ribe</a> • <a href=3D"https://info.yahoo.com/legal/us/yahoo/utos/terms= /" style=3D"text-decoration: none;">Terms of Use</a> </div> </div> <br> <!-- |**|end egp html banner|**| --> </div> <!-- ygrp-msg --> =20 <!-- Sponsor --> <!-- |**|begin egp html banner|**| --> <div id=3D"ygrp-sponsor" style=3D"width:160px; float:right; clear:none; m= argin:0 0 25px 0; background: #fff;"> <!-- Start Recommendations --> <div id=3D"ygrp-reco"> </div> <!-- End Recommendations --> </div> <!-- |**|end egp html banner|**| --> <div style=3D"clear:both; color: #FFF; font-size:1px;">.</div> </div> <img src=3D"http://geo.yahoo.com/serv?s=3D97359714/grpId=3D1522270/grpspI= d=3D1705006764/msgId=3D1927/stime=3D1593506795" width=3D"1" height=3D"1"> <= br> <img src=3D"http://y.analytics.yahoo.com/fpc.pl?ywarid=3D515FB27823A7407E&a= =3D10001310322279&js=3Dno&resp=3Dimg" width=3D"1" height=3D"1">=20 <div style=3D"color: #fff; height: 0;">__,_._,___</div> <!--~-|**|PrettyHtmlEnd|**|-~--> </body> <!--~-|**|PrettyHtmlStart|**|-~--> <head> <style type=3D"text/css"> <!-- #ygrp-mkp { border: 1px solid #d8d8d8; font-family: Arial; margin: 10px 0; padding: 0 10px; } #ygrp-mkp hr { border: 1px solid #d8d8d8; } #ygrp-mkp #hd { color: #628c2a; font-size: 85%; font-weight: 700; line-height: 122%; margin: 10px 0; } #ygrp-mkp #ads { margin-bottom: 10px; } #ygrp-mkp .ad { padding: 0 0; } #ygrp-mkp .ad p { margin: 0; } #ygrp-mkp .ad a { color: #0000ff; text-decoration: none; } #ygrp-sponsor #ygrp-lc { font-family: Arial; } #ygrp-sponsor #ygrp-lc #hd { margin: 10px 0px; font-weight: 700; font-size: 78%; line-height: 122%; } #ygrp-sponsor #ygrp-lc .ad { margin-bottom: 10px; padding: 0 0; } #actions { font-family: Verdana; font-size: 11px; padding: 10px 0; } #activity { background-color: #e0ecee; float: left; font-family: Verdana; font-size: 10px; padding: 10px; } #activity span { font-weight: 700; } #activity span:first-child { text-transform: uppercase; } #activity span a { color: #5085b6; text-decoration: none; } #activity span span { color: #ff7900; } #activity span .underline { text-decoration: underline; } .attach { clear: both; display: table; font-family: Arial; font-size: 12px; padding: 10px 0; width: 400px; } .attach div a { text-decoration: none; } .attach img { border: none; padding-right: 5px; } .attach label { display: block; margin-bottom: 5px; } .attach label a { text-decoration: none; } =20=20 blockquote { margin: 0 0 0 4px; } .bold { font-family: Arial; font-size: 13px; font-weight: 700; } .bold a { text-decoration: none; } dd.last p a { font-family: Verdana; font-weight: 700; } dd.last p span { margin-right: 10px; font-family: Verdana; font-weight: 700; } dd.last p span.yshortcuts { margin-right: 0; } div.attach-table div div a { text-decoration: none; } div.attach-table { width: 400px; } div.file-title a, div.file-title a:active, div.file-title a:hover, div.fi= le-title a:visited { text-decoration: none; } div.photo-title a, div.photo-title a:active, div.photo-title a:hover, div= ..photo-title a:visited { text-decoration: none; } div#ygrp-mlmsg #ygrp-msg p a span.yshortcuts { font-family: Verdana; font-size: 10px; font-weight: normal; } .green { color: #628c2a; } .MsoNormal { margin: 0 0 0 0; } o { font-size: 0; } #photos div { float: left; width: 72px; } #photos div div { border: 1px solid #666666; height: 62px; overflow: hidden; width: 62px; } #photos div label { color: #666666; font-size: 10px; overflow: hidden; text-align: center; white-space: nowrap; width: 64px; } #reco-category { font-size: 77%; } #reco-desc { font-size: 77%; } .replbq { margin: 4px; } #ygrp-actbar div a:first-child { /* border-right: 0px solid #000;*/ margin-right: 2px; padding-right: 5px; } #ygrp-mlmsg { font-size: 13px; font-family: Arial, helvetica,clean, sans-serif; *font-size: small; *font: x-small; } #ygrp-mlmsg table { font-size: inherit; font: 100%; } #ygrp-mlmsg select, input, textarea { font: 99% Arial, Helvetica, clean, sans-serif; } #ygrp-mlmsg pre, code { font:115% monospace; *font-size:100%; } #ygrp-mlmsg * { line-height: 1.22em; } #ygrp-mlmsg #logo { padding-bottom: 10px; } #ygrp-msg p a { font-family: Verdana; } #ygrp-msg p#attach-count span { color: #1E66AE; font-weight: 700; } #ygrp-reco #reco-head { color: #ff7900; font-weight: 700; } #ygrp-reco { margin-bottom: 20px; padding: 0px; } #ygrp-sponsor #ov li a { font-size: 130%; text-decoration: none; } #ygrp-sponsor #ov li { font-size: 77%; list-style-type: square; padding: 6px 0; }=20 #ygrp-sponsor #ov ul { margin: 0; padding: 0 0 0 8px; } #ygrp-text { font-family: Georgia; } #ygrp-text p { margin: 0 0 1em 0; } #ygrp-text tt { font-size: 120%; } #ygrp-vital ul li:last-child { border-right: none !important;=20 }=20 --> </style> </head> <!--~-|**|PrettyHtmlEnd|**|-~--> </html> <!-- end group email --> --Ov15EKkbhJrWkjqP37tT1iYZjJ5FqltEevItD0S--