Re: encoding of SF files, allowable characters

Grove, Michael wrote:
 > smartfrog gurus,
 >
 > i've just started reading through smartfrog documentation and have been
 > playing with the latest distribution on sourceforge.  i'm still
 > recovering from I18N testing that i performed on a previous project, so
 > you'll understand my initial questions being on that topic.
 >
 > i couldn't find a discussion in the documentation on the expected
 > encoding of SF files.  is the file assumed to be UTF-8?  can that
 > encoding be modified, either globally or on a per file basis?  i'm not
 > requesting that it be configurable, just want to understand how things
 > work today.
 >
 > related to that question is one on allowable characters in SF files.  in
 > particular, i'm wondering whether there are restrictions on allowable
 > characters for attribute names and (more importantly) attribute values,
 > or whether all unicode characters are supported.  i also would be
 > interested in knowing if there are restrictions on component names,
 > include paths, codebase paths, etc.
 >
 > sorry if i missed this in the docs - i'd appreciate a pointer if i did.
 >
 > thanks.
 >

Ah, now that's a really interesting question.

If you look at the javacc parser, it looks like you can have octal 
characters such as \277 , but only three byte ones.

The parser itself feeds input streams (or unicode Reader instances) 
through another generated class, a JavaCharStream which is described as :

/**
  * An implementation of interface CharStream, where the stream is 
assumed to
  * contain only ASCII characters (with java-like unicode escape 
processing).
  */

So
  1. ASCII only
  2. octal characters are allowed
  3. unicode chars are inserted with \u0234 style insertions, very much 
like Java itself.


Because that unicode expansion kicks in before the parser gets to see 
anything, it looks like you can insert unicode in anywhere you like, 
possibly even in the middle of a component name or a reference. They 
work inside values too, though you'd expect something like a double 
quote " to need escaping there.

Once actually deployed, both attribute names and values can be of any 
object type you like, as long as it serializes. In the (prototype) CDL 
XML language, we had full unicode support, using the xml1.0 element 
naming rules to set elements; the values coming from nested text. when 
deployed, all namespaced attributes were published as XML qualified 
names, a (uri, localname) pair that could not be referenced from 
smartfrog at all. That is: the set of names and values you can use as 
attributes is much bigger than the .sf files themselves can handle.


1. If this is not the I18n behaviour you want, let patrick know!

2. we dont have any explicit i18n tests right now. We test on the 
various systems we have to hand, not in any odd locales (turkey, french 
canadian) and so don't catch the surprises they raise. I worry about the 
number of places the code probably has String.equalsIgnoreCase() rather 
than a String.toLower(LOCALE_EN).equals(other.toLower(LOCALE_EN)) 
comparison. We should do an audit for that and fix all occurrences. 
Checkstyle could look for us.


-steve

-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys - and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV