Re: PingTool.py converts 8 bit characters into entities

Hi!

I just dug a bit deeper, use this script to reproduce. You need strong
nerves to read on. You have been warned.

Python 2.2 (#1, 11/12/02, 23:31:59) 
[GCC Apple cpp-precomp 6.14] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import xmlrpclib
>>> srv = xmlrpclib.Server('http://rpc.weblogs.com/RPC2')
>>> ord('ä')                         
228
>>> srv._ServerProxy__verbose = 1
>>> srv.weblogUpdates.ping('ä2', 'http://localhost/x2')

You need to change the blog title and url if you make several tries one
after the other, as weblogs.com only accepts a ping every 30 minutes and
so effectively makes it very hard to debug using your actual software
with your actual data ...

I send in the following request:

<?xml version='1.0'?>
<methodCall>
<methodName>weblogUpdates.ping</methodName>
<params>
<param>
<value><string>&amp;#228;2</string></value>
</param>
<param>
<value><string>http://localhost/x2</string></value>
</param>
</params>
</methodCall>

This is perfectly legal us-ascii encoded stuff. The &auml; is encoded as
entity &#228; and this is quoted in the string value, as should be done
with XMLRPC. I don't declare a charset, neither at HTTP nor at XML
level. So it should be regarded as utf-8 if you follow the XML spec.

This is what I get back. Note that in the headers there is _no_ charset
declaration, neither is at the xml level! (formatting with tabs is
removed - Userland products go to great length to correctly format and
indent XML source, but don't get the charset right. Go figure.)

header: Connection: close
header: Content-Length: 501
header: Content-Type: text/xml
header: Date: Wed, 17 Mar 2004 10:24:02 GMT
header: Server: UserLand Frontier/9.0-WinNT
body:

<?xml version="1.0"?>
<methodResponse>
<params>
<param>
<value><struct>
<member>
<name>flerror</name>
<value>
<boolean>0</boolean>
</value>
</member>
<member>
<name>message</name>
<value>Thanks for the ping. We checked and found that the "\xe42" weblog
has changed, so it will appear in &lt;a
href="http://www.weblogs.com/changes.xml">changes.xml&lt;/a> next time
it is updated.</value>
</member>
</struct></value>
</param>
</params>
</methodResponse>

The \xe4 actually is the directly encoded "ä" character. Yes, the one I
passed in as &#228;!

So what they do: they decode the entity encoded &#XXX; to their
ISO-8859-1 encoding and store that in their database. They return the
decoded entity as a ISO-8859-1 character - without declaring their
charset in the XML header! Of course the Expat parser barfs on that one.
Had they just returned what I sent them, everything would be fine, as
that would still be a vaild utf-8 encoded beast (as us-ascii is a valid
subset of utf-8). But no, they need to decode my request to deliberately
break the response for XML parsers.

This is so stupid ...

With utf-8 encoded strings it actually works - kind of. XMLRPC is happy,
as they pass the utf-8 stuff out in the same way as it was passed in.
But it won't show up on their homepage correctly, as they send out utf-8
stuff directly byte-per-byte in an otherwise ISO-8859-1 encoded page. So
actually you see garbage. You see much garbage on <http://weblogs.com/>
nowadays, as many japanese and other languages are used in weblogs.
Weblogs.com _must_ switch over to storing everything as utf-8 internally
and rendering pages in utf-8 encoding. At least if they still want to be
the prime weblog notification service.

unicode entities (entities with character codes above 255) are stripped
of their & and ; and are returned (and supposedly stored) as just the
number. At least that's what a quick test showed me. So they still
interpret stuff, but don't let it pass on untouched, but deliberately
break it. Talk about stupid ...

bye, Georg