Re: byte order marker puzzle

I'll track this issue here: https://github.com/shirok/Gauche/issues/966


On Tue, Dec 5, 2023 at 6:00 PM Shiro Kawai <[email protected]> wrote:

> Oh, it may cause an issue.  As the legacy code opens a port with
> open-input-port, even if it wants to read binary.  Dropping the beginning
> EF BB BF would disturb such code.  BOM-dropping operation should be opt-in
> to keep backward compatibility.
>
> On Tue, Dec 5, 2023 at 5:57 PM Shiro Kawai <[email protected]> wrote:
>
>> Your assessment is correct.  The file contains UTF-8 BOM, and Gauche does
>> not treat it specially, so the first string is actually "\xfeff;1".  Using
>> BOM in utf-8 text stream is not recommended by the unicode standard, but
>> there are programs that generate them, so we should deal with it.
>>
>> For typical usage, the Right Thing may be to drop BOM when the file is
>> opened as a textual port, while keeping it if the file is opened as a
>> binary port.  It may cause some confusion, for Gauche hasn't been treating
>> textual port and binary port differently, following Unix's "everything is a
>> byte stream" philosophy.  It may be the time to move on, though.
>>
>> On the side note, I think 'write' should print U+FEFF (Zero width
>> no-break space) in escaped format, so that it is easier to find the
>> character within a string.
>>
>>
>>
>>
>> On Tue, Dec 5, 2023 at 3:40 PM Stephen Lewis <[email protected]>
>> wrote:
>>
>>> Shiro,
>>> I am having trouble with a text file that has a BOM
>>> (byte order mark) at the beginning. 'read-line'
>>> reads a string which appears to contain a number
>>> 'print' can print the number, but 'string->number'
>>> returns '#f'. It seems as if the BOM is somehow
>>> visible to 'string->number'.
>>> Here is a short test program and text file that
>>> shows the problem. Is this expected behaviour of
>>> 'read-line' and/or 'string->number'? Many programs
>>> treat the BOM like a magic number and read the
>>> file after the BOM. Am I doing something wrong?
>>> Stephen Lewis
>>>
>>> ====================================================
>>> ;;;
>>> ;;; Odd behaviour of 'string->number' if BOM is present
>>> ;;;
>>> ;;; Stephen Lewis 3-dec 2023
>>> ;;;
>>>
>>> (define (usage program-name)
>>>   (format (current-error-port)
>>>           "Usage: gosh ~a ./bom.txt\n" program-name)
>>>   (exit 2))
>>>
>>> (define (do-test args)
>>>   (let1 fn (cadr args)
>>>         (print "Input file \"" (string-scan-right fn "/" 'after) "\"")
>>>         (call-with-input-file fn get-numbers)
>>>         ))
>>>
>>> (define (get-numbers port)
>>> (until (read-line port) eof-object? => line
>>>     (print "Line contains \"" line "\"")
>>>     (print "Line is a string? " (string? line))
>>>     (print "Parses as number " (string->number line))))
>>>
>>> (define (main args)
>>> (if (eq? (length args) 2)
>>>   (do-test args)
>>>   (usage (car args)))
>>> 0)
>>> ====================================================
>>>
>>> This generates the test input file:
>>> echo "77u/MQ0KMg0KMw0K" | base64 -d > bom.txt
>>>
>>> Gauche version:
>>> gosh -V
>>> Gauche scheme shell, version 0.9.12 [utf-8,pthreads], x86_64-pc-linux-gnu
>>> (version "0.9.12")
>>> (command "gosh")
>>> (scheme.id gauche)
>>> (languages scheme r5rs r7rs)
>>> (encodings utf-8)
>>> (website "https://practical-scheme.net/gauche")
>>> (build.platform "x86_64-pc-linux-gnu")
>>> (build.configure "--prefix=/usr" "--build=x86_64-pc-linux-gnu"
>>> "--host=x86_64-pc-linux-gnu" "--mandir=/usr/share/man"
>>> "--infodir=/usr/share/info" "--datadir=/usr/share" "--sysconfdir=/etc"
>>> "--localstatedir=/var/lib" "--datarootdir=/usr/share"
>>> "--docdir=/usr/share/doc/gauche-0.9.12"
>>> "--htmldir=/usr/share/doc/gauche-0.9.12/html" "--libdir=/usr/lib64"
>>> "--enable-ipv6" "--with-ca-bundle=/etc/ssl/certs/ca-certificates.crt"
>>> "--with-slib=/usr/share/slib" "--with-tls=mbedtls"
>>> "build_alias=x86_64-pc-linux-gnu" "host_alias=x86_64-pc-linux-gnu"
>>> "CFLAGS=-O2 -pipe -mtune=native -march=native" "LDFLAGS=-Wl,-O1
>>> -Wl,--as-needed" "--enable-multibyte=utf-8")
>>> (scheme.path "/usr/share/gauche-0.98/site/lib"
>>> "/usr/share/gauche-0.98/0.9.12/lib")
>>> (gauche.threads pthreads)
>>> (gauche.net.tls mbedtls)
>>>
>>>
>>>
>>>
>>>
>>> _______________________________________________
>>> Gauche-devel mailing list
>>> [email protected]
>>> https://lists.sourceforge.net/lists/listinfo/gauche-devel
>>>
>>

_______________________________________________
Gauche-devel mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/gauche-devel