[ sisc-Bugs-1391498 ] Character streams are not UTF-8 by default, contrary to docs

Bugs item #1391498, was opened at 2005-12-27 22:02
Message generated for change (Tracker Item Submitted) made by Item Submitter
You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=379534&aid=1391498&group_id=23735

Please note that this message will contain a full copy of the comment thread,
including the initial issue submission, for this request,
not just the latest update.
Category: None
Group: None
Status: Open
Resolution: None
Priority: 5
Submitted By: Norman Gray (normang)
Assigned to: Nobody/Anonymous (nobody)
Summary: Character streams are not UTF-8 by default, contrary to docs

Initial Comment:
The `Dynamic Parameters' section of the manual says that SISC's 
character streams use UTF8 by default.  This appears not to be true.

[I get confused by character encodings, so I'm willing to be persuaded 
I'm misunderstanding something significant here]

I've attached a zip file containing:

    utf8.patch
        A patch against the SISC CVS HEAD, which makes
        UTF-8 the default character encoding

    sisc-chars-u and sisc-chars.in-u
        a SISC script and input file, encoded as UTF-8

    sisc-chars-i and sisc-chars.in-i
        the same, but converted to ISO-8859-1

    run-tests.sh
        a shell script which runs the two sisc-chars-? scripts
        with different inputs

    sisc-1.12.0-beta
        a directory containing the results of running
        run-tests.sh using SISC 1.12.0-beta

    sisc-patched
        the same, using the patched SISC

The contents of sisc-1.12.0-beta are, I believe, garbled in various 
ways.

First, in the first two cases run by run-tests.sh, the script reports the 
character-set parameter to be MacRoman (I'm running on OSX), rather 
than UTF-8, which the manual suggests should be the default.

The four cases in the sisc-chars-? scripts are

1: 8-bit characters included in script, no files read, and no
   character sets specified (ie, all the defaults)
2: some 8-bit characters from file, no character sets specified
3: output encodings specified (in with-output-to-file)
4: input and output encodings specified (in with-output-to-file
   and with-input-from-file)

The run-tests.sh script runs one or other of the sisc-chars-? scripts, 
with or without a character set specified in sisc.characterSet.

When using SISC 1.12.0-beta, some of the output files should be 
garbled (see below), but others are garbled when they should not be:

-- uuu-sisc-chars.out.{1,2}: the literal characters are OK, but the 
\u00E9 escape is garbled; ...{3,4} are just generally garbled

-- The uui-* and uii-* cases are variously garbled.

-- The iii-* cases are correct, which is reasonable, since in this case 
everything is specified.

That is, it appears that (i) the `the default character set used by 
character ports' is not UTF-8, despite the statement that it is in the 
`Dynamic Parameters' section, and (ii) the \u escape appears not to be 
working.  I think this is just because the default is not being forced to 
be UTF-8, and is instead defaulting to MacRoman (which is probably a 
separately bad thing, but not our problem).  The cause of part of this 
is, I think, that sisc.SourceInputPort is in several places created with a 
simple InputStream, rather than a Reader, and is separately linked to 
the definition of getDefaultCharacterSet in 
sisc.env.DynamicEnvironment.

When using the patched SISC:

-- The four uuu-* cases should be correct and identical, and are so.

-- Of the uui-* files, only uui-sisc-chars.out.4 is garbled, correctly, 
since in this case the UTF-8 file sisc-chars.in-u is being read in as 
ISO-8859-1.

-- Of the uii-* case, the literal characters in the script are garbled, 
correctly (since in this case the UTF-8 script file sisc-chars-u is being 
read as ISO-8859-1), but the \u00E9 escape, and the characters read 
from the file, are being interpreted correctly.

-- The iii-* files are all correct and identical.


I've taken the liberty of creating and including a patch, against the 
current CVS HEAD, which effects these changes.  The patch should be 
self-explanatory.  It has the effect of making UTF-8 the default 
character encoding for both character streams and the Scheme 
program source itself.

It also changes DynamicEnvironment.java so that the characterSet field 
is a java.nio.charset.Charset, rather than a String, and adds a String-
>Charset converter in Util.java, which keeps the various Charset 
Exceptions in one place, and implements the Charset defaulting 
conveniently.  The former involves some refactorings elsewhere, such 
as in Networking.java. 

The DynamicEnvironment constructors are modified so that the 
WriterOutputPort has an explicit character set.  This means that 
(display) within a script (as opposed to within the REPL) displays 8-bit 
characters correctly.  This appeared to work before, though I'm now 
not sure why (!), but stopped working with my patch, until I made this 
change.

Minor fix: the table in the `Dynamic Parameters' section of the manual 
is slightly garbled, as the `Scheme parameters' column is missing for 
the first two rows.  That's also fixed in the patch.


----------------------------------------------------------------------

You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=379534&aid=1391498&group_id=23735


-------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc. Do you grep through log files
for problems?  Stop!  Download the new AJAX search engine that makes
searching your log files as easy as surfing the  web.  DOWNLOAD SPLUNK!
http://ads.osdn.com/?ad_id=7637&alloc_id=16865&op=click