Re: Unicode characters and htmlfill issue

Hi Marius,
thank you very much for your support both on IRC and here in the list.  
I've done some experiments with String() and UnicodeString()  
validators and I think to have thoroughly understood your explanation  
of what is going on.

However, during these experiments, I've found a little glitch in the  
String() validator behaviour. As you pointed out String() leaves the  
strings untouched. In fact, this is not completely true. Just try:

assert String().to_python(u'') is u''
assert String().from_python(u'') is u''

You get in both cases an assertion exception since, both calls to .to/ 
from_python() returns '' (and not u'' as expected).

I agree that returning an empty basestring instead of the empty  
unicode string is not a great problem. However, I still think it  
should be fixed anyway. Any comment?

Cheers,
  Andrea


On 25 Mar, 2009, at 14:37, Marius Gedminas wrote:

> Hi again!
>
> On Mon, Mar 23, 2009 at 06:43:00PM +0100, Andrea Riciputi wrote:
>> I have a problem with UnicodeString validator and htmlfill.
>
> We discussed this on IRC
> (http://pylonshq.com/irclogs/%23pylons/%23pylons.2009-03-24.log.html#t2009-03-24T09 
> :04:00
> is the horrible url) and decided that the right solution is to use
> String validators of formencode.
>
> The difference between String and UnicodeString is that String passes
> through str and unicode objects unchanged, while UnicodeString always
> ensures you get unicode from to_python() and always returns str from
> from_python().
>
> Pylons already takes care of str <-> unicode conversions for us,  
> both on
> the input side and on the output side, so having FormEncode do it  
> again
> causes problems.
>
> Here's my recommendations for avoiding Unicode problems:
>
>  * make sure you get unicode objects from the database (i.e. use
>    SQLAlchemy's UnicodeString columns)
>  * always use unicode objects internally (pure-ASCII str objects don't
>    hurt, so pure ASCII string constants in the source code are fine --
>    but when you're taking user input, make sure to convert it to
>    unicode as soon as possible)
>  * do not use formencode.validators.UnicodeString
>  * pass unicode strings to htmlfill.render()
>
> This way SQLAlchemy takes care of conversion at the database side,  
> Pylons
> takes care of conversion at the HTML side, and you always deal with
> unicode strings.
>
> Now, I've only been using Pylons for two months so I'm not a great
> expert.  If there are any holes in this logic, please poke at them!
>
> Marius Gedminas
> -- 
> The irony is that Bill Gates claims to be making a stable operating
> system and Linus Torvalds claims to be trying to take over the
> world.
> 		-- seen on the net
> ------------------------------------------------------------------------------
> Apps built with the Adobe(R) Flex(R) framework and Flex Builder(TM)  
> are
> powering Web 2.0 with engaging, cross-platform capabilities. Quickly  
> and
> easily build your RIAs with Flex Builder, the Eclipse(TM)based  
> development
> software that enables intelligent coding and step-through debugging.
> Download the free 60 day trial. http://p.sf.net/sfu/www-adobe-com_______________________________________________
> FormEncode-discuss mailing list
> FormEncode-discuss-5NWGOfrQmneRv+LV9MX5uipxlwaOVQ5f@public.gmane.org
> https://lists.sourceforge.net/lists/listinfo/formencode-discuss


------------------------------------------------------------------------------