Re: byte order marker puzzle

Shiro,
Thanks for (speedy) analysis.
It seems to be an ugly problem. As I understand it
the BOM is "optional" and one may encounter five
different BOM signatures:

00 00 FE FF 	UTF-32, big-endian
FF FE 00 00 	UTF-32, little-endian
FE FF 	UTF-16, big-endian
FF FE 	UTF-16, little-endian
EF BB BF 	UTF-8

My test file has the 3-byte BOM for UTF-8.

Ideally I would like to parse the file and get the
same output for input with or without the BOM.
This issue is not really a gauche/Scheme problem but
I was fooled because 'cat file-with-BOM' and
Gauche 'print' _appeared_ to print a valid number.
The BOM seems to show as a zero width character with
no visible output but I have discovered that
both 'cat' and 'print' do in fact output the BOM
bytes. Further 'read-line' packs the line containing
a BOM and a digit into a string of length 2, the
first character being invisible. 'read-line' seems
to put the BOM bytes into the string whether or not
'allow-byte-string?' is set, presumably because the
BOM sequence is a valid though invisible UTF character.
Since the BOM signatures are never useful inside a file maybe they
should not be placed in a string as a character at all.
'string-size' however reports size as 4 and 'string-byte-ref'
gives access to all 4 bytes of the line.
Curiously, although 'od -t x1 bom.txt' shows "ef bb bf 31 0a"
'vi' does not show the BOM unless the same three bytes
occur later in the file when 'vi' shows "<feff>" which is,
as you said a zero width space, but _not_ the bytes "ef bb bf"
(a different zero length space??) which are actually in the file. 
I also note that a BOM signature on the front of a script
prevents "#!/bin/bash" or "#!/usr/bin/gosh" from being
recognized and hence the script does not run. So BOM
also fools the Linux loader!
You have a typo in issue "UTF8 BOM handing" s/handing/handling/.

Thanks for help,
Stephen Lewis

On Tue, 5 Dec 2023 17:57:06 -1000
Shiro Kawai <[email protected]> wrote:

> Your assessment is correct.  The file contains UTF-8 BOM, and Gauche does
> not treat it specially, so the first string is actually "\xfeff;1".  Using
> BOM in utf-8 text stream is not recommended by the unicode standard, but
> there are programs that generate them, so we should deal with it.
> 
> For typical usage, the Right Thing may be to drop BOM when the file is
> opened as a textual port, while keeping it if the file is opened as a
> binary port.  It may cause some confusion, for Gauche hasn't been treating
> textual port and binary port differently, following Unix's "everything is a
> byte stream" philosophy.  It may be the time to move on, though.
> 
> On the side note, I think 'write' should print U+FEFF (Zero width no-break
> space) in escaped format, so that it is easier to find the character within
> a string.


-- 
Stephen Lewis <[email protected]>