Re: byte order marker puzzle

Your assessment is correct.  The file contains UTF-8 BOM, and Gauche does
not treat it specially, so the first string is actually "\xfeff;1".  Using
BOM in utf-8 text stream is not recommended by the unicode standard, but
there are programs that generate them, so we should deal with it.

For typical usage, the Right Thing may be to drop BOM when the file is
opened as a textual port, while keeping it if the file is opened as a
binary port.  It may cause some confusion, for Gauche hasn't been treating
textual port and binary port differently, following Unix's "everything is a
byte stream" philosophy.  It may be the time to move on, though.

On the side note, I think 'write' should print U+FEFF (Zero width no-break
space) in escaped format, so that it is easier to find the character within
a string.




On Tue, Dec 5, 2023 at 3:40 PM Stephen Lewis <[email protected]>
wrote:

> Shiro,
> I am having trouble with a text file that has a BOM
> (byte order mark) at the beginning. 'read-line'
> reads a string which appears to contain a number
> 'print' can print the number, but 'string->number'
> returns '#f'. It seems as if the BOM is somehow
> visible to 'string->number'.
> Here is a short test program and text file that
> shows the problem. Is this expected behaviour of
> 'read-line' and/or 'string->number'? Many programs
> treat the BOM like a magic number and read the
> file after the BOM. Am I doing something wrong?
> Stephen Lewis
>
> ====================================================
> ;;;
> ;;; Odd behaviour of 'string->number' if BOM is present
> ;;;
> ;;; Stephen Lewis 3-dec 2023
> ;;;
>
> (define (usage program-name)
>   (format (current-error-port)
>           "Usage: gosh ~a ./bom.txt\n" program-name)
>   (exit 2))
>
> (define (do-test args)
>   (let1 fn (cadr args)
>         (print "Input file \"" (string-scan-right fn "/" 'after) "\"")
>         (call-with-input-file fn get-numbers)
>         ))
>
> (define (get-numbers port)
> (until (read-line port) eof-object? => line
>     (print "Line contains \"" line "\"")
>     (print "Line is a string? " (string? line))
>     (print "Parses as number " (string->number line))))
>
> (define (main args)
> (if (eq? (length args) 2)
>   (do-test args)
>   (usage (car args)))
> 0)
> ====================================================
>
> This generates the test input file:
> echo "77u/MQ0KMg0KMw0K" | base64 -d > bom.txt
>
> Gauche version:
> gosh -V
> Gauche scheme shell, version 0.9.12 [utf-8,pthreads], x86_64-pc-linux-gnu
> (version "0.9.12")
> (command "gosh")
> (scheme.id gauche)
> (languages scheme r5rs r7rs)
> (encodings utf-8)
> (website "https://practical-scheme.net/gauche")
> (build.platform "x86_64-pc-linux-gnu")
> (build.configure "--prefix=/usr" "--build=x86_64-pc-linux-gnu"
> "--host=x86_64-pc-linux-gnu" "--mandir=/usr/share/man"
> "--infodir=/usr/share/info" "--datadir=/usr/share" "--sysconfdir=/etc"
> "--localstatedir=/var/lib" "--datarootdir=/usr/share"
> "--docdir=/usr/share/doc/gauche-0.9.12"
> "--htmldir=/usr/share/doc/gauche-0.9.12/html" "--libdir=/usr/lib64"
> "--enable-ipv6" "--with-ca-bundle=/etc/ssl/certs/ca-certificates.crt"
> "--with-slib=/usr/share/slib" "--with-tls=mbedtls"
> "build_alias=x86_64-pc-linux-gnu" "host_alias=x86_64-pc-linux-gnu"
> "CFLAGS=-O2 -pipe -mtune=native -march=native" "LDFLAGS=-Wl,-O1
> -Wl,--as-needed" "--enable-multibyte=utf-8")
> (scheme.path "/usr/share/gauche-0.98/site/lib"
> "/usr/share/gauche-0.98/0.9.12/lib")
> (gauche.threads pthreads)
> (gauche.net.tls mbedtls)
>
>
>
>
>
> _______________________________________________
> Gauche-devel mailing list
> [email protected]
> https://lists.sourceforge.net/lists/listinfo/gauche-devel
>

_______________________________________________
Gauche-devel mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/gauche-devel