Re: Gelex with Unicode
"Eric Bezault ericb-D6Qt/9opevxWk0Htik3J/[email protected] [gobo-eiffel]" <[email protected]> Tue, 30 Jun 2020 12:04:39 +0200
| Newsgroups | gmane.comp.lang.eiffel.gobo.general |
|---|---|
| Message-ID | <[email protected]> |
--mPbSdeq1IkWkqEYXioO9hq91lNA5PfYB27efnTh Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit On 30/06/2020 11:20, Alexander Kogtenkov kwaxer-JGs/[email protected] [gobo-eiffel] wrote: > YY_UNICODE_FILE_BUFFER reads data into a buffer with 32-bit characters. > I.e., we have the following sequence of steps: > Step 1. Read unchanged data from a file into an intermediate buffer. > Step 2. Convert the intermediate buffer to UTF-32 and store it in buffer > with 32-bit characters. > Step 3. Let a scanner read from the buffer with 32-bit characters. > What I have in mind is: > Step 1. Read unchanged data from a file into an intermediate buffer. > Step 2. Let a scanner retrieve a 32-bit character. The character is > computed at this point from the intermediate buffer. > I.e. the scanner does not read from the pre-converted buffer, but asks > to read from the unchanged original data. OK, I see what you mean. The scanner engine is optimized to read data as expected from its own buffer. That's why, even prior to supporting Unicode, the scanner does not read from the FILE.last_string (or directly from a string if one want to parse a string rather than a file) but copies the data to be scanned to its own buffer even when no conversion was needed. Furthermore, the scanner may read, unread, read again, read several characters forward, step back etc. We would have to have a test each time it needs to access a character: has this character already been built from its encoding or not. That's already two reasons for performance decrease. The only benefit that I can think off is when we want to stop scanning of a file well before the end of the file. Do you have cases like that? If it is just to read the few first tokens to find for example the name of the class at the beginning of an Eiffel file, I would recommend create a input file buffer with a smaller size to limit the amount of data to be read from the file. But just in that case, and keep a large buffer size when parsing the entire class file. Another solution could be to use the interactive mode when generating the scanner with gelex. But that would be very slow I think. It's mainly used when reading from the console while the user is typing. > > > Side note. It looks like setting the default encoding does not > affect whether BOM is processed or not. I.e., if the default > encoding is set to ISO 8859-1, the buffers still read and interpret > BOM as if no encoding was specified. So, if a file in ISO 8859-1 > starts with a BOM, it would be interpreted as UTF-8 with the BOM > removed. Maybe, I’m wrong here, but this is what I inferred from the > code. Whether this is "a bug or a feature", I do not know, but this > may need to be either fixed or documented. > > If I remember currently the default encoding is used when no BOM is > found in the file. It does not override a BOM if found in the file. > > So, if an ISO 8859-1 file starts with a BOM, it is always interpreted as > UTF-8, and never as ISO 8859-1. But what are the bytes in this BOM? Aren't they the bytes indicating that the file is actually encoded with UTF-8? Are you saying that you want the bytes of the BOM to be read as CHARACTER_8 characters using ISO 8859-1? If that's what you want to do, i.e. read as ISO 8859-1 despite the BOM, you should use YY_FILE_BUFFER. But if you just want to discard the BOM, and make as if it was not there and consider the following characters as ISO 8859-1 that the BOM was using UTF-8, then there is no class in Gobo to do that. But you can write your own descendant of YY_BUFFER to do that. If it's something to be added to the existing classes, I think that `default_encoding` alone is not enough. We should have something like `bom_to_be_discarded` so that when the scanner reads a BOM, it just discards its bytes. From there the scanner will fall back to the `default_encoding` to decide which encoding to use instead of the BOM which has just been discarded. -- Eric Bezault mailto:ericb-D6Qt/9opevxWk0Htik3J/[email protected] http://www.gobosoft.com --mPbSdeq1IkWkqEYXioO9hq91lNA5PfYB27efnTh Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN" "http://www.w3.org/TR/htm= l4/strict.dtd"> <html> <head> </head> =20 <body style=3D"background-color: #fff;"> <span style=3D"display:none"> </span> <!--~-|**|PrettyHtmlStartT|**|-~--> <div id=3D"ygrp-mlmsg" style=3D"position:relative;"> <div id=3D"ygrp-msg" style=3D"z-index: 1;"> <!--~-|**|PrettyHtmlEndT|**|-~--> <div id=3D"ygrp-text" > =20=20=20=20=20=20 =20=20=20=20=20=20 <p>On 30/06/2020 11:20, Alexander Kogtenkov kwaxer-JGs/[email protected] [gobo-eiff= el] wrote:<br> > YY_UNICODE_FILE_BUFFER reads data into a buffer with 32-bit characters= .. <br> > I.e., we have the following sequence of steps:<br> > Step 1. Read unchanged data from a file into an intermediate buffer.<b= r> > Step 2. Convert the intermediate buffer to UTF-32 and store it in buff= er <br> > with 32-bit characters.<br> > Step 3. Let a scanner read from the buffer with 32-bit characters.<br> > What I have in mind is:<br> > Step 1. Read unchanged data from a file into an intermediate buffer.<b= r> > Step 2. Let a scanner retrieve a 32-bit character. The character is <b= r> > computed at this point from the intermediate buffer.<br> > I.e. the scanner does not read from the pre-converted buffer, but asks= <br> > to read from the unchanged original data.<br> <br> OK, I see what you mean. The scanner engine is optimized to read<br> data as expected from its own buffer. That's why, even prior to<br> supporting Unicode, the scanner does not read from the FILE.last_string<br> (or directly from a string if one want to parse a string rather than<br> a file) but copies the data to be scanned to its own buffer even when<br> no conversion was needed. Furthermore, the scanner may read, unread,<br> read again, read several characters forward, step back etc. We would<br> have to have a test each time it needs to access a character: has<br> this character already been built from its encoding or not. That's<br> already two reasons for performance decrease. The only benefit that<br> I can think off is when we want to stop scanning of a file well before<br> the end of the file. Do you have cases like that? If it is just to read<br> the few first tokens to find for example the name of the class at the<br> beginning of an Eiffel file, I would recommend create a input file<br> buffer with a smaller size to limit the amount of data to be read<br> from the file. But just in that case, and keep a large buffer size<br> when parsing the entire class file.<br> <br> Another solution could be to use the interactive mode when generating<br> the scanner with gelex. But that would be very slow I think. It's<br> mainly used when reading from the console while the user is typing.<br> <br> > <br> > > Side note. It looks like setting the default encoding does n= ot<br> > affect whether BOM is processed or not. I.e., if the default<br> > encoding is set to ISO 8859-1, the buffers still read and interpre= t<br> > BOM as if no encoding was specified. So, if a file in ISO 8859-1<b= r> > starts with a BOM, it would be interpreted as UTF-8 with the BOM<b= r> > removed. Maybe, I=E2=80=99m wrong here, but this is what I inferre= d from the<br> > code. Whether this is "a bug or a feature", I do not kno= w, but this<br> > may need to be either fixed or documented.<br> > <br> > If I remember currently the default encoding is used when no BOM i= s<br> > found in the file. It does not override a BOM if found in the file= ..<br> > <br> > So, if an ISO 8859-1 file starts with a BOM, it is always interpreted = as <br> > UTF-8, and never as ISO 8859-1.<br> <br> But what are the bytes in this BOM? Aren't they the bytes indicating<br= > that the file is actually encoded with UTF-8? Are you saying that<br> you want the bytes of the BOM to be read as CHARACTER_8 characters<br> using ISO 8859-1? If that's what you want to do, i.e. read as<br> ISO 8859-1 despite the BOM, you should use YY_FILE_BUFFER. But if<br> you just want to discard the BOM, and make as if it was not there<br> and consider the following characters as ISO 8859-1 that the BOM<br> was using UTF-8, then there is no class in Gobo to do that. But you<br> can write your own descendant of YY_BUFFER to do that.<br> <br> If it's something to be added to the existing classes, I think that<br> `default_encoding` alone is not enough. We should have something like<br> `bom_to_be_discarded` so that when the scanner reads a BOM, it just<br> discards its bytes. From there the scanner will fall back to the<br> `default_encoding` to decide which encoding to use instead of the<br> BOM which has just been discarded.<br> <br> -- <br> Eric Bezault<br> mailto:ericb-D6Qt/9opevxWk0Htik3J/[email protected]<br> http://www.gobosoft.com<br> </p> </div> =20=20=20=20=20 <!--~-|**|PrettyHtmlStart|**|-~--> <div style=3D"color: #fff; height: 0;">__._,_.___</div> =20=20=20=20=20=20=20=20=20=20 =20=20 =20 =20=20=20=20 <div style=3D"clear:both"> </div> <div id=3D"fromDMARC" style=3D"margin-top: 10px;"> <hr style=3D"height:2px ; border-width:0; color:#E3E3E3; background= -color:#E3E3E3;"> Posted by: Eric Bezault <ericb-D6Qt/9opevxWk0Htik3J/[email protected]> <hr style= =3D"height:2px ; border-width:0; color:#E3E3E3; background-color:#E3E3E3;"> </div> <div style=3D"clear:both"> </div> <table cellspacing=3D4px style=3D"margin-top: 10px; margin-bottom: 10px= ; color: #2D50FD;"> <tbody> <tr> <td style=3D"font-size: 12px; font-family: arial; font-weight: bo= ld; padding: 7px 5px 5px;" > <a style=3D"text-decoration: none; color: #2D50FD= " href=3D"https://groups.yahoo.com/neo/groups/gobo-eiffel/conversations/mes= sages/1929;_ylc=3DX3oDMTJwa24yc3N0BF9TAzk3MzU5NzE0BGdycElkAzE1MjIyNzAEZ3Jwc= 3BJZAMxNzA1MDA2NzY0BG1zZ0lkAzE5MjkEc2VjA2Z0cgRzbGsDcnBseQRzdGltZQMxNTkzNTEx= NDgz?act=3Dreply&messageNum=3D1929">Reply via web post</a> </td> <td>•</td> <td style=3D"font-size: 12px; font-family: arial; padding: 7px 5p= x 5px;" > <a href=3D"mailto:ericb-D6Qt/9opevxWk0Htik3J/[email protected]?subject=3DRe%3A%20%5Bgobo-= eiffel%5D%20Gelex%20with%20Unicode" style=3D"text-decoration: none; color: = #2D50FD;"> Reply to sender </a> </td> <td>•</td> <td style=3D"font-size: 12px; font-family: arial; padding: 7px 5p= x 5px;"> <a href=3D"mailto:[email protected]?subject=3DRe%3A%2= 0%5Bgobo-eiffel%5D%20Gelex%20with%20Unicode" style=3D"text-decoration: none= ; color: #2D50FD"> Reply to group </a> </td> <td>•</td> <td style=3D"font-size: 12px; font-family: arial; padding: 7px 5p= x 5px;" > <a href=3D"https://groups.yahoo.com/neo/groups/gobo-eiffel/conv= ersations/newtopic;_ylc=3DX3oDMTJlb2dlODlzBF9TAzk3MzU5NzE0BGdycElkAzE1MjIyN= zAEZ3Jwc3BJZAMxNzA1MDA2NzY0BHNlYwNmdHIEc2xrA250cGMEc3RpbWUDMTU5MzUxMTQ4Mw--= " style=3D"text-decoration: none; color: #2D50FD">Start a New Topic</a> </td> <td>•</td> <td style=3D"font-size: 12px; font-family: arial; padding: 7px 5p= x 5px;color: #2D50FD;" > <a href=3D"https://groups.yahoo.com/neo/groups/= gobo-eiffel/conversations/topics/1923;_ylc=3DX3oDMTM0bm9iZTU3BF9TAzk3MzU5Nz= E0BGdycElkAzE1MjIyNzAEZ3Jwc3BJZAMxNzA1MDA2NzY0BG1zZ0lkAzE5MjkEc2VjA2Z0cgRzb= GsDdnRwYwRzdGltZQMxNTkzNTExNDgzBHRwY0lkAzE5MjM-" style=3D"text-decoration: = none; color: #2D50FD;">Messages in this topic</a> (7) </td> </tr> </tbody> </table> =20=20=20=20=20=20=20=20 <!------- Start Nav Bar ------> <!-- |**|begin egp html banner|**| --> <!-- |**|end egp html banner|**| --> <div id=3D"ygrp-grfd" style=3D"font-family: Verdana; font-size: 12px; p= adding: 15px 0;"> =20=20=20=20=20=20 <!-- |**|begin egp html banner|**| --> To Post a message, send it to: [email protected]<BR= > To Unsubscribe, send a blank message to: gobo-eiffel-unsubscribe-B11MqFFcr06IwRZHo2/[email protected]= m=20=20=20=20=20=20 <!-- |**|end egp html banner|**| --> </div> =20=20 =20 <!-- |**|begin egp html banner|**| --> <div id=3D"ygrp-vital" style=3D"background-color: #f2f2f2; font-family: Ver= dana; font-size: 10px; margin-bottom: 10px; padding: 10px;"> <span id=3D"vithd" style=3D"font-weight: bold; color: #333; text-transf= orm: uppercase; "><a href=3D"https://groups.yahoo.com/neo/groups/gobo-eiffe= l/info;_ylc=3DX3oDMTJlYjdjNXJpBF9TAzk3MzU5NzE0BGdycElkAzE1MjIyNzAEZ3Jwc3BJZ= AMxNzA1MDA2NzY0BHNlYwN2dGwEc2xrA3ZnaHAEc3RpbWUDMTU5MzUxMTQ4Mw--" style=3D"t= ext-decoration: none;">Visit Your Group</a></span> <ul style=3D"list-style-type: none; margin: 0; padding: 0; display: in= line;"> </ul> </div> <div id=3D"ft" style=3D"font-family: Arial; font-size: 11px; margin-top: 5p= x; padding: 0 2px 0 0; clear: both;"> <a href=3D"https://groups.yahoo.com/neo;_ylc=3DX3oDMTJkNjJ1ZTFvBF9TAzk3Mz= U5NzE0BGdycElkAzE1MjIyNzAEZ3Jwc3BJZAMxNzA1MDA2NzY0BHNlYwNmdHIEc2xrA2dmcARzd= GltZQMxNTkzNTExNDgz" style=3D"float: left;"><img src=3D"https://s.yimg.com/= ru/0.9.12/min/css/yahoo_en-US_f_pw_101x21.png" height=3D"21" width=3D"101" = alt=3D"Yahoo! Groups" style=3D"border: 0;"/></a> <div style=3D"color: #747575; float: right;"> • <a href=3D"https://i= nfo.yahoo.com/privacy/us/yahoo/groups/details.html" style=3D"text-decoratio= n: none;">Privacy</a> • <a href=3D"mailto:gobo-eiffel-unsubscribe@yaho= ogroups.com?subject=3DUnsubscribe" style=3D"text-decoration: none;">Unsubsc= ribe</a> • <a href=3D"https://info.yahoo.com/legal/us/yahoo/utos/terms= /" style=3D"text-decoration: none;">Terms of Use</a> </div> </div> <br> <!-- |**|end egp html banner|**| --> </div> <!-- ygrp-msg --> =20 <!-- Sponsor --> <!-- |**|begin egp html banner|**| --> <div id=3D"ygrp-sponsor" style=3D"width:160px; float:right; clear:none; m= argin:0 0 25px 0; background: #fff;"> <!-- Start Recommendations --> <div id=3D"ygrp-reco"> </div> <!-- End Recommendations --> </div> <!-- |**|end egp html banner|**| --> <div style=3D"clear:both; color: #FFF; font-size:1px;">.</div> </div> <img src=3D"http://geo.yahoo.com/serv?s=3D97359714/grpId=3D1522270/grpspI= d=3D1705006764/msgId=3D1929/stime=3D1593511483" width=3D"1" height=3D"1"> <= br> <img src=3D"http://y.analytics.yahoo.com/fpc.pl?ywarid=3D515FB27823A7407E&a= =3D10001310322279&js=3Dno&resp=3Dimg" width=3D"1" height=3D"1">=20 <div style=3D"color: #fff; height: 0;">__,_._,___</div> <!--~-|**|PrettyHtmlEnd|**|-~--> </body> <!--~-|**|PrettyHtmlStart|**|-~--> <head> <style type=3D"text/css"> <!-- #ygrp-mkp { border: 1px solid #d8d8d8; font-family: Arial; margin: 10px 0; padding: 0 10px; } #ygrp-mkp hr { border: 1px solid #d8d8d8; } #ygrp-mkp #hd { color: #628c2a; font-size: 85%; font-weight: 700; line-height: 122%; margin: 10px 0; } #ygrp-mkp #ads { margin-bottom: 10px; } #ygrp-mkp .ad { padding: 0 0; } #ygrp-mkp .ad p { margin: 0; } #ygrp-mkp .ad a { color: #0000ff; text-decoration: none; } #ygrp-sponsor #ygrp-lc { font-family: Arial; } #ygrp-sponsor #ygrp-lc #hd { margin: 10px 0px; font-weight: 700; font-size: 78%; line-height: 122%; } #ygrp-sponsor #ygrp-lc .ad { margin-bottom: 10px; padding: 0 0; } #actions { font-family: Verdana; font-size: 11px; padding: 10px 0; } #activity { background-color: #e0ecee; float: left; font-family: Verdana; font-size: 10px; padding: 10px; } #activity span { font-weight: 700; } #activity span:first-child { text-transform: uppercase; } #activity span a { color: #5085b6; text-decoration: none; } #activity span span { color: #ff7900; } #activity span .underline { text-decoration: underline; } .attach { clear: both; display: table; font-family: Arial; font-size: 12px; padding: 10px 0; width: 400px; } .attach div a { text-decoration: none; } .attach img { border: none; padding-right: 5px; } .attach label { display: block; margin-bottom: 5px; } .attach label a { text-decoration: none; } =20=20 blockquote { margin: 0 0 0 4px; } .bold { font-family: Arial; font-size: 13px; font-weight: 700; } .bold a { text-decoration: none; } dd.last p a { font-family: Verdana; font-weight: 700; } dd.last p span { margin-right: 10px; font-family: Verdana; font-weight: 700; } dd.last p span.yshortcuts { margin-right: 0; } div.attach-table div div a { text-decoration: none; } div.attach-table { width: 400px; } div.file-title a, div.file-title a:active, div.file-title a:hover, div.fi= le-title a:visited { text-decoration: none; } div.photo-title a, div.photo-title a:active, div.photo-title a:hover, div= ..photo-title a:visited { text-decoration: none; } div#ygrp-mlmsg #ygrp-msg p a span.yshortcuts { font-family: Verdana; font-size: 10px; font-weight: normal; } .green { color: #628c2a; } .MsoNormal { margin: 0 0 0 0; } o { font-size: 0; } #photos div { float: left; width: 72px; } #photos div div { border: 1px solid #666666; height: 62px; overflow: hidden; width: 62px; } #photos div label { color: #666666; font-size: 10px; overflow: hidden; text-align: center; white-space: nowrap; width: 64px; } #reco-category { font-size: 77%; } #reco-desc { font-size: 77%; } .replbq { margin: 4px; } #ygrp-actbar div a:first-child { /* border-right: 0px solid #000;*/ margin-right: 2px; padding-right: 5px; } #ygrp-mlmsg { font-size: 13px; font-family: Arial, helvetica,clean, sans-serif; *font-size: small; *font: x-small; } #ygrp-mlmsg table { font-size: inherit; font: 100%; } #ygrp-mlmsg select, input, textarea { font: 99% Arial, Helvetica, clean, sans-serif; } #ygrp-mlmsg pre, code { font:115% monospace; *font-size:100%; } #ygrp-mlmsg * { line-height: 1.22em; } #ygrp-mlmsg #logo { padding-bottom: 10px; } #ygrp-msg p a { font-family: Verdana; } #ygrp-msg p#attach-count span { color: #1E66AE; font-weight: 700; } #ygrp-reco #reco-head { color: #ff7900; font-weight: 700; } #ygrp-reco { margin-bottom: 20px; padding: 0px; } #ygrp-sponsor #ov li a { font-size: 130%; text-decoration: none; } #ygrp-sponsor #ov li { font-size: 77%; list-style-type: square; padding: 6px 0; }=20 #ygrp-sponsor #ov ul { margin: 0; padding: 0 0 0 8px; } #ygrp-text { font-family: Georgia; } #ygrp-text p { margin: 0 0 1em 0; } #ygrp-text tt { font-size: 120%; } #ygrp-vital ul li:last-child { border-right: none !important;=20 }=20 --> </style> </head> <!--~-|**|PrettyHtmlEnd|**|-~--> </html> <!-- end group email --> --mPbSdeq1IkWkqEYXioO9hq91lNA5PfYB27efnTh--