Re: byte order marker puzzle

Oh, it may cause an issue.  As the legacy code opens a port with
open-input-port, even if it wants to read binary.  Dropping the beginning
EF BB BF would disturb such code.  BOM-dropping operation should be opt-in
to keep backward compatibility.

On Tue, Dec 5, 2023 at 5:57 PM Shiro Kawai <[email protected]> wrote:

> Your assessment is correct.  The file contains UTF-8 BOM, and Gauche does
> not treat it specially, so the first string is actually "\xfeff;1".  Using
> BOM in utf-8 text stream is not recommended by the unicode standard, but
> there are programs that generate them, so we should deal with it.
>
> For typical usage, the Right Thing may be to drop BOM when the file is
> opened as a textual port, while keeping it if the file is opened as a
> binary port.  It may cause some confusion, for Gauche hasn't been treating
> textual port and binary port differently, following Unix's "everything is a
> byte stream" philosophy.  It may be the time to move on, though.
>
> On the side note, I think 'write' should print U+FEFF (Zero width no-break
> space) in escaped format, so that it is easier to find the character within
> a string.
>
>
>
>
> On Tue, Dec 5, 2023 at 3:40 PM Stephen Lewis <[email protected]>
> wrote:
>
>> Shiro,
>> I am having trouble with a text file that has a BOM
>> (byte order mark) at the beginning. 'read-line'
>> reads a string which appears to contain a number
>> 'print' can print the number, but 'string->number'
>> returns '#f'. It seems as if the BOM is somehow
>> visible to 'string->number'.
>> Here is a short test program and text file that
>> shows the problem. Is this expected behaviour of
>> 'read-line' and/or 'string->number'? Many programs
>> treat the BOM like a magic number and read the
>> file after the BOM. Am I doing something wrong?
>> Stephen Lewis
>>
>> ====================================================
>> ;;;
>> ;;; Odd behaviour of 'string->number' if BOM is present
>> ;;;
>> ;;; Stephen Lewis 3-dec 2023
>> ;;;
>>
>> (define (usage program-name)
>>   (format (current-error-port)
>>           "Usage: gosh ~a ./bom.txt\n" program-name)
>>   (exit 2))
>>
>> (define (do-test args)
>>   (let1 fn (cadr args)
>>         (print "Input file \"" (string-scan-right fn "/" 'after) "\"")
>>         (call-with-input-file fn get-numbers)
>>         ))
>>
>> (define (get-numbers port)
>> (until (read-line port) eof-object? => line
>>     (print "Line contains \"" line "\"")
>>     (print "Line is a string? " (string? line))
>>     (print "Parses as number " (string->number line))))
>>
>> (define (main args)
>> (if (eq? (length args) 2)
>>   (do-test args)
>>   (usage (car args)))
>> 0)
>> ====================================================
>>
>> This generates the test input file:
>> echo "77u/MQ0KMg0KMw0K" | base64 -d > bom.txt
>>
>> Gauche version:
>> gosh -V
>> Gauche scheme shell, version 0.9.12 [utf-8,pthreads], x86_64-pc-linux-gnu
>> (version "0.9.12")
>> (command "gosh")
>> (scheme.id gauche)
>> (languages scheme r5rs r7rs)
>> (encodings utf-8)
>> (website "https://practical-scheme.net/gauche")
>> (build.platform "x86_64-pc-linux-gnu")
>> (build.configure "--prefix=/usr" "--build=x86_64-pc-linux-gnu"
>> "--host=x86_64-pc-linux-gnu" "--mandir=/usr/share/man"
>> "--infodir=/usr/share/info" "--datadir=/usr/share" "--sysconfdir=/etc"
>> "--localstatedir=/var/lib" "--datarootdir=/usr/share"
>> "--docdir=/usr/share/doc/gauche-0.9.12"
>> "--htmldir=/usr/share/doc/gauche-0.9.12/html" "--libdir=/usr/lib64"
>> "--enable-ipv6" "--with-ca-bundle=/etc/ssl/certs/ca-certificates.crt"
>> "--with-slib=/usr/share/slib" "--with-tls=mbedtls"
>> "build_alias=x86_64-pc-linux-gnu" "host_alias=x86_64-pc-linux-gnu"
>> "CFLAGS=-O2 -pipe -mtune=native -march=native" "LDFLAGS=-Wl,-O1
>> -Wl,--as-needed" "--enable-multibyte=utf-8")
>> (scheme.path "/usr/share/gauche-0.98/site/lib"
>> "/usr/share/gauche-0.98/0.9.12/lib")
>> (gauche.threads pthreads)
>> (gauche.net.tls mbedtls)
>>
>>
>>
>>
>>
>> _______________________________________________
>> Gauche-devel mailing list
>> [email protected]
>> https://lists.sourceforge.net/lists/listinfo/gauche-devel
>>
>

_______________________________________________
Gauche-devel mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/gauche-devel